open-r1deepseek-r1grporeinforcement-learninghugging-facerayvllmtutorial

Open-R1 Guide: Reproducing DeepSeek-R1 with GRPO & Ray

Open-R1 Architecture: Hugging Face Open-R1 Pipeline, Distributed GRPO on Ray and vLLM, Verifiable Rewards, and Reasoning Distillation

The Problem: The Open Reproduction Void of Frontier Reasoning Models

When DeepSeek published the technical report for DeepSeek-R1 and released its model weights, it marked a historic inflection point in artificial intelligence. By applying pure Reinforcement Learning (RL) directly to a base language model, DeepSeek demonstrated that complex multi-step reasoning, self-reflection, backtracking, and algorithmic verification could emerge naturally without massive human-annotated chain-of-thought datasets.

In autonomous data extraction pipelines, enterprise research copilots, and desktop AI engines like AiFetchly, reasoning models represent a quantum leap in capability. They allow agents to autonomously debug failing web scrapers, synthesize messy financial filings, and verify complex mathematical schemas without user intervention.

However, the wider AI community and enterprise engineering teams quickly encountered a major roadblock: the open reproduction void.

The Open-Weights Paradox in Frontier Reasoning: ┌─────────────────────────────────────────────────────────────────┐ │ Released Artifacts: Model Weights (DeepSeek-R1, Distilled R1) │ ├─────────────────────────────────────────────────────────────────┤ │ Missing Foundations: │ │ - Zero open training code or distributed RL orchestration scripts│ │ - Undisclosed synthetic data curation & filtering pipelines │ │ - Undocumented prompt formatting & reward function thresholds │ │ - Opaque compute cluster recipes (vLLM, Ray, DeepSpeed configs) │ └─────────────────────────────────────────────────────────────────┘ Result: The community can RUN R1, but cannot TRAIN or CUSTOMIZE it.

1. The Closed Reality of “Open Weights” Reasoning Checkpoints

While open weights allow developers to serve models locally, they do not democratize the scientific method:

  • No Training Recipes: The community was left guessing how DeepSeek calibrated cold-start Supervised Fine-Tuning (SFT), how reward functions prevented language-mixing and format hacking, and how hyperparameter schedules stabilized long-sequence RL.
  • Inability to Adapt to Novel Domains: Enterprise teams cannot adapt reasoning capabilities to private proprietary datasets (such as internal ERP systems, proprietary medical diagnostic protocols, or novel programming languages) without a repeatable post-training RL pipeline.

2. The Infrastructure Bottleneck: Decoupling Rollout Generation from Backprop

Training reasoning models introduces a severe distributed systems bottleneck that standard training frameworks (like DeepSpeed or Megatron) were never designed to solve: the Rollout-Training Decoupling problem.

During Reinforcement Learning with Group Relative Policy Optimization (GRPO):

  1. The system must generate $G$ candidate reasoning traces (rollouts) for every prompt, often spanning 4,096 to 16,384 tokens of detailed exploration.
  2. Generating these rollouts requires inference-optimized kernels (PagedAttention, dynamic batching, continuous scheduling).
  3. Once generated, computing policy gradients and updating model weights requires training-optimized distributed frameworks (FSDP2, DeepSpeed ZeRO-3, gradient accumulation).

Traditional RLHF tools collocated training and inference on the same GPU workers, resulting in massive GPU idle bubbles and catastrophic training slowdowns.

3. The Cold-Start Data Scarcity and Verification Crisis

A naive base model exposed to pure RL without guidance frequently degenerates into repetitive loops, format collapse, or multilingual gibberish before discovering effective reasoning patterns.

DeepSeek avoided this using a curated cold-start SFT dataset (thousands of readable, high-quality reasoning traces) paired with rule-based verifiers. However, open-source datasets lacked verified, machine-checkable solutions for complex reasoning tasks, creating a chicken-and-egg problem for independent researchers.

Open-R1 Architecture: The Three-Stage Reproduction Blueprint

To bridge this gap, Hugging Face launched Open-R1—a transparent, community-driven initiative engineered to reproduce the entire R1 reasoning pipeline from scratch using open data, open code, and open compute.

The Open-R1 Three-Stage Pipeline Architecture: ┌────────────────────────────────────────────────────────────────────────┐ │ Stage 1: Cold-Start SFT Curation │ │ - Extract open math/code problems (NuminaMath, OlympiadBench, Codeforces│ │ - Synthetic reasoning generation via R1 & Qwen-2.5-72B-Instruct │ │ - Strict Math-Verify & AST execution filtering -> 15k clean traces │ └───────────────────────────────────┬────────────────────────────────────┘ │ (Cold-Start SFT Checkpoint) ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ Stage 2: Large-Scale Distributed GRPO Training │ │ - Decoupled Architecture: vLLM on Ray (Rollouts) + TRL FSDP2 (Trainer) │ │ - Verifiable Reward Functions (Zero Critic Network, 0 extra VRAM) │ │ - Format & Language Constraints (Tags: <think>...</think>, No mixing) │ └───────────────────────────────────┬────────────────────────────────────┘ │ (Open-R1 Zero/Full Model) ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ Stage 3: Rejection Sampling & Knowledge Distillation │ │ - Sample 800k rollouts across diverse reasoning prompts │ │ - Keep only verified correct traces with efficient reasoning steps │ │ - Fine-tune lightweight student models (Llama-3.1 8B, Qwen-2.5 7B/14B) │ └───────────────────────────────────┬────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ High-Accuracy On-Device & Cloud Reasoning Checkpoints (100% Open) │ └────────────────────────────────────────────────────────────────────────┘

1. Stage 1: Cold-Start SFT Curation with Open Synthetic Data

Rather than relying on proprietary annotations, Open-R1 constructs a clean, reproducible cold-start dataset:

  • Source Corpus Ingestion: Aggregates math and code problems from verified open sources (NuminaMath-CoT, OlympiadBench, APPS, LiveCodeBench, and GSM8K).
  • Synthetic Trace Generation: Prompts high-capacity frontier open models (such as Qwen-2.5-72B-Instruct and DeepSeek-R1) to generate long-chain-of-thought derivations enclosed within <think> ... </think> delimiters.
  • Automated Verification Gate: Every generated trace is evaluated through deterministic execution engines:
    • Symbolic mathematical verification via Math-Verify (parsing LaTeX answers, handling permutations, fractions, and symbolic algebraic equivalence).
    • Sandboxed Python execution checking unit test pass rates for coding challenges.
  • Traces that arrive at the wrong answer or fail formatting constraints are discarded immediately. The resulting dataset (Open-R1-Math-220k) serves as the open foundation for SFT.

2. Stage 2: Large-Scale Distributed GRPO Training with TRL & Ray

The intellectual core of Open-R1 is its distributed reinforcement learning architecture, implemented via Hugging Face TRL (Transformer Reinforcement Learning):

  • Critic-Free Advantage Estimation: Open-R1 replaces memory-hungry Actor-Critic PPO with Group Relative Policy Optimization (GRPO). For each training prompt $q$, the policy generates a cohort of $G = 8$ or $16$ candidate completions.
  • Rule-Based Reward Verifiers: Candidate outputs are scored against objective verification functions:

$$r_i = r_{\text{accuracy}} + r_{\text{format}} + r_{\text{language}}$$

  • Dynamic Normalization: The relative advantage $A_i$ for each rollout is calculated directly against the mean and variance of its cohort:

$$A_i = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r}) + \epsilon}$$

Because no separate Critic network is instantiated, training VRAM is slashed by 50%, enabling teams to train 32B and 70B reasoning models on standard academic and enterprise clusters.

3. Stage 3: Rejection Sampling and Knowledge Distillation

To bring reasoning capabilities to edge devices, local copilots, and lightweight automation engines like AiFetchly, Open-R1 implements high-throughput distillation:

  • Massive Rejection Sampling: Generates over 800,000 reasoning trajectories across diverse domains using the Stage 2 model.
  • Strict Quality Filtering: Retains only traces where the model arrived at the ground-truth solution, executed valid self-correction steps, and avoided redundant rambling.
  • Student SFT: Fine-tunes compact base architectures (such as Qwen-2.5-7B or Llama-3.1-8B) on these verified reasoning traces, producing lightweight models that outperform GPT-4o on competitive benchmarks like MATH-500.

Technical Mechanics: Distributed Rollouts, Verifiers, and Loss Formulations

Reproducing DeepSeek-R1 at scale requires rigorous systems engineering to coordinate asynchronous inference generation with synchronous distributed backpropagation.

Open-R1 Distributed Execution Topology: ┌────────────────────────────────────────────────────────────────────────┐ │ Ray Orchestrator Cluster │ ├──────────────────────────────────┬─────────────────────────────────────┤ │ Inference Workers (vLLM Engine) │ Training Workers (TRL + FSDP2) │ ├──────────────────────────────────┼─────────────────────────────────────┤ │ - Node 1-4: 32x H100 GPUs │ - Node 5-8: 32x H100 GPUs │ │ - Continuous PagedAttention │ - Full Sharded Data Parallel (FSDP2)│ │ - High-throughput batched rollouts│ - Forward pass on sampled groups │ │ - Emits G=8 trajectories/prompt │ - Computes clipped GRPO gradients │ └─────────────────┬────────────────┴──────────────────┬──────────────────┘ │ │ ▼ ▼ Shared Memory Plasma Store ◄────────── Weight Synchronization (Fast zero-copy trajectory transfer) (Broadcast updated weights)

1. The Distributed Ray + vLLM Generation Engine

In standard Hugging Face training loops, generating $G = 8$ rollouts of length 8,192 using standard PyTorch generate() is painfully slow.

Open-R1 decouples the pipeline using Ray and vLLM:

  • Dedicated Rollout Pool: A dedicated subset of GPU nodes runs vLLM instances equipped with PagedAttention, chunked prefill, and dynamic scheduling.
  • Zero-Copy Trajectory Transfer: Sampled tokens, log probabilities, and prompt tensors are written to Ray’s shared memory Plasma store.
  • Weight Synchronization: After every training step, updated model weights from the training cluster are asynchronously broadcast to the vLLM rollout workers via high-speed NCCL or distributed disk checkpoints, keeping the generation policy aligned with parameter updates.

2. Verifiable Reward Functions: Math-Verify and Sandboxed Code Execution

A foundational discovery of R1 is that reward models are prone to reward hacking—models learn to output flowery prose that tricks a neural judge without solving the underlying logic.

Open-R1 mandates deterministic, rule-based verification:

# Open-R1 Verifiable Reward Function Implementation import re from math_verify import parse, verify def accuracy_reward_func(completions, solution, **kwargs): """Calculates binary 0/1 accuracy using symbolic Math-Verify.""" rewards = [] for completion, sol in zip(completions, solution): # Extract content outside thinking tags answer_match = re.search(r"</think>\s*(.*)", completion, re.DOTALL) candidate = answer_match.group(1) if answer_match else completion # Parse and symbolically compare mathematical expressions gold_parsed = parse(sol) candidate_parsed = parse(candidate) is_correct = verify(gold_parsed, candidate_parsed) rewards.append(1.0 if is_correct else 0.0) return rewards def format_reward_func(completions, **kwargs): """Enforces strict reasoning delimiters without format drift.""" pattern = r"^<think>\n.*?\n</think>\n.*" return [1.0 if re.match(pattern, c, re.DOTALL) else 0.0 for c in completions]

By awarding positive rewards only when the symbolic expression matches ground truth and the formatting strictly adheres to <think> ... </think>, the policy is physically incapable of reward hacking.

3. Length-Normalized GRPO Objective and KL Drift Penalties

To prevent the model from artificially padding responses to exploit reward biases, Open-R1 formulates the GRPO surrogate objective with an analytical KL divergence penalty:

$$\mathcal{J}{\text{GRPO}}(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum{i=1}^G \left( \min \left( \frac{\pi_\theta(o_i|q)}{\pi_{\text{old}}(o_i|q)} A_i, ; \text{clip}\left(\frac{\pi_\theta(o_i|q)}{\pi_{\text{old}}(o_i|q)}, 1-\epsilon, 1+\epsilon\right) A_i \right) - \beta D_{\text{KL}}(\pi_\theta ,||, \pi_{\text{ref}}) \right) \right]$$

  • Token-Level Loss Normalization: Gradients are normalized across valid completion tokens to prevent long reasoning traces from dominating batch updates.
  • Analytical KL Estimate: Rather than sampling an external distribution, KL divergence is calculated directly from policy log-ratios, eliminating unnecessary computation.

4. Hardware Sizing and Multi-Node Cluster Economics

Open-R1 Hardware & Scaling Reference Table: ┌──────────────────────┬──────────────┬──────────────┬──────────────────┐ │ Model Scale │ Cluster Size │ Training │ Estimated Cloud │ │ │ │ Duration │ Compute Cost │ ├──────────────────────┼──────────────┼──────────────┼──────────────────┤ │ Open-R1-Zero 1.5B │ 8x H100 GPUs │ ~18 Hours │ ~$450 │ │ Open-R1-Zero 7B │ 32x H100 GPUs│ ~36 Hours │ ~$3,800 │ │ Open-R1-Zero 14B │ 64x H100 GPUs│ ~48 Hours │ ~$9,800 │ │ Open-R1-Full 32B │ 128x H100s │ ~72 Hours │ ~$32,000 │ └──────────────────────┴──────────────┴──────────────┴──────────────────┘

By eliminating the Critic model and accelerating rollouts with vLLM, reproducing reasoning capabilities on a 7B model costs under $4,000—well within reach of academic labs and growth-stage AI startups.

Empirical Benchmarks: Open-R1 Models vs. DeepSeek-R1 Baselines

Hugging Face’s Open-R1 models were evaluated against proprietary frontiers and DeepSeek checkpoints across three rigorous reasoning benchmarks: AIME 2024 (High-School Olympiad Math), MATH-500 (Advanced Mathematics), and LiveCodeBench (Competitive Coding).

Reasoning Benchmark Accuracy (% Correct): ┌──────────────────────────────┬──────────────┬──────────────┬──────────────────┐ │ Model Architecture │ AIME 2024 │ MATH-500 │ LiveCodeBench │ │ │ (Pass@1) │ (Pass@1) │ (Pass@1) │ ├──────────────────────────────┼──────────────┼──────────────┼──────────────────┤ │ Base Model (Qwen-2.5-7B) │ 15.6% │ 52.4% │ 18.2% │ │ DeepSeek-R1-Distill-Qwen-7B │ 55.5% │ 92.8% │ 37.6% │ │ Open-R1-Distill-7B (HF) │ 54.2% │ 91.5% │ 36.8% │ │ Open-R1-Zero-7B (Pure RL) │ 46.8% │ 86.4% │ 32.1% │ │ GPT-4o (Closed Frontier API) │ 38.3% │ 74.6% │ 33.8% │ └──────────────────────────────┴──────────────┴──────────────┴──────────────────┘

1. Benchmark Analysis

  • Distillation Fidelity: Open-R1-Distill-7B achieves 91.5% on MATH-500 and 54.2% on AIME 2024, closely matching DeepSeek’s official distillation checkpoint (92.8% and 55.5%) and outperforming base GPT-4o by a wide margin.
  • Pure RL Emergence: Even without human cold-start traces, Open-R1-Zero-7B demonstrates the self-correction phenomenon, improving its math score from 52.4% to 86.4% through reward-driven trial and error.

2. Training Throughput and Rollout Efficiency Scaling

Using the Ray + vLLM decoupled architecture, Open-R1 achieves an average of 285 tokens/second per GPU during rollout generation, a 4.2x increase over collocated PyTorch eager sampling.

Production Implementation: Training Reasoning Models with Open-R1

Here is the step-by-step engineering workflow to configure verifiable reward functions, launch distributed GRPO training with Hugging Face TRL, and deploy distilled reasoning models.

Step 1: Environment Setup and Open-R1 Installation

Clone the official Open-R1 repository and install the verified training dependencies:

# Clone the open-r1 repository git clone https://github.com/huggingface/open-r1.git cd open-r1 # Install core dependencies with TRL, vLLM, and flash-attention pip install -e . pip install vllm ray math-verify trl>=0.14.0

Step 2: Configuring Custom Verifiable Reward Functions

Define domain-specific verifiable reward evaluators in Python:

import re from math_verify import parse, verify def build_combined_reward_evaluator(): """Combines rule-based verification for math and strict syntax formatting.""" def compute_rewards(prompts, completions, solution, **kwargs): scores = [] for completion, gold in zip(completions, solution): reward = 0.0 # 1. Format Verification: Must contain closed think tags if "<think>" in completion and "</think>" in completion: reward += 0.2 else: scores.append(0.0) continue # 2. Extract final derivation after thinking tags parts = completion.split("</think>") if len(parts) >= 2: final_answer = parts[-1].strip() # 3. Deterministic math equivalence check try: if verify(parse(gold), parse(final_answer)): reward += 0.8 except Exception: pass scores.append(reward) return scores return compute_rewards

Step 3: Launching Distributed GRPO Training on Ray and vLLM

Create the training launch configuration (configs/grpo_qwen_7b.yaml):

# Open-R1 GRPO Configuration model_name_or_path: 'Qwen/Qwen2.5-Math-7B' dataset_name: 'open-r1/OpenR1-Math-220k' dataset_configs: 'default' # Hyperparameters learning_rate: 2.0e-5 per_device_train_batch_size: 2 gradient_accumulation_steps: 4 num_generations: 8 # G = 8 rollouts per prompt max_prompt_length: 512 max_completion_length: 4096 temperature: 0.7 # Rollout Engine use_vllm: true vllm_device: 'auto' vllm_gpu_memory_utilization: 0.7 # Output & Logging output_dir: './output/open-r1-qwen-7b' logging_steps: 5 save_strategy: 'steps' save_steps: 100

Launch the distributed training script:

# Launch multi-GPU GRPO training using Hugging Face Accelerate and Ray accelerate launch \ --config_file configs/accelerate_configs/zero3.yaml \ src/open_r1/grpo.py \ --config configs/grpo_qwen_7b.yaml

Step 4: Deploying Open-R1 Distilled Models in aiFetchly Workflows

In local desktop automation and autonomous extraction pipelines like AiFetchly, deploying a fine-tuned Open-R1 student model provides high-level reasoning for complex scraping and schema verification without cloud API dependencies:

from vllm import LLM, SamplingParams class LocalReasoningExtractor: """Local reasoning agent powered by Open-R1 distilled checkpoints.""" def __init__(self, model_path: str = "open-r1/Open-R1-Distill-Qwen-7B"): self.llm = LLM( model=model_path, tensor_parallel_size=1, gpu_memory_utilization=0.85, max_model_len=8192 ) self.sampling_params = SamplingParams( temperature=0.6, top_p=0.95, max_tokens=4096 ) def extract_and_verify(self, html_snippet: str, schema: str) -> str: prompt = f"""<|im_start|>system You are an expert data verification agent. Reason through the input HTML and schema inside <think>...</think> tags, then output the verified JSON.<|im_end|> <|im_start|>user Schema: {schema} HTML Content: {html_snippet}<|im_end|> <|im_start|>assistant <think> """ outputs = self.llm.generate([prompt], self.sampling_params) return outputs[0].outputs[0].text # Usage in aiFetchly local extraction loop # extractor = LocalReasoningExtractor() # result = extractor.extract_and_verify(raw_table_html, target_json_schema)

Summary & Architectural Best Practices for Open Reasoning Systems

Hugging Face’s Open-R1 initiative democratizes post-training reasoning, demonstrating that frontier intelligence is an engineering discipline governed by verifiable data and distributed policy optimization rather than proprietary black boxes.

Open-R1 Architecture Decision Matrix: Project Objective & Resource Budget: ├─ Rapid Local Prototyping & Domain Exploration (< 24GB GPU) │ └─► Recommendation: Serve Open-R1-Distill-7B via vLLM; skip RL training ├─ Enterprise Domain Specialization (Internal Schemas, Financial Logic) │ └─► Recommendation: Stage 1 SFT on curated open traces + domain rejection sampling ├─ Full Autonomous Reasoning Emergence (Academic / Frontier R&D) │ └─► Recommendation: Distributed GRPO on Ray + vLLM with deterministic verifiers └─ Edge Ingestion & Automated Extraction (<InternalLink feature="homepage">aiFetchly</InternalLink> Workflows) └─► Recommendation: Quantized 4-bit Open-R1 Distill running locally on CPU/GPU

Architectural Takeaways

  1. Rely on Deterministic Verifiers, Not Neural Judges: Model-based reward estimators invite reward hacking. Build verifiable reward functions using AST execution, symbolic parsers (Math-Verify), and strict formatting regexes.
  2. Decouple Rollouts from Gradient Updates: Never run serial PyTorch sampling during large-scale GRPO. Use Ray and vLLM to scale rollout throughput to hundreds of tokens per second per GPU.
  3. Reasoning Distillation Is Unusually Efficient: Small student models (7B to 8B) fine-tuned on verified reasoning trajectories retain over 95% of frontier reasoning performance at a fraction of inference cost.