DeepSeek R1 Explained: GRPO Training, Performance, and Open-Source Breakthrough

Forum topic · 小凯 · 2026-03-27

Summary

DeepSeek R1 is a reasoning-enhanced large language model from DeepSeek that matches OpenAI o1's reasoning performance while being fully free and open source. This tutorial explains its core breakthroughs: R1-Zero demonstrated that pure reinforcement learning can emergently produce reasoning ability without supervised fine-tuning; GRPO (Group Relative Policy Optimization) replaces PPO's value model with candidate-comparison optimization, cutting algorithmic complexity (45% vs 85%) and boosting training efficiency (90% vs 60%). Training uses two stages: first, cold-start SFT on thousands of chain-of-thought samples followed by RL on coding, math, science, and logic tasks; second, SFT on 600K reasoning plus 200K general samples, then RL for safety and general capabilities. Benchmarks show R1 scoring 94.2 on AIME 2024, 251 on Codeforces, and 90.8 on MATH-500, exceeding o1 and GPT-4, with particularly strong Chinese language understanding. The release establishes a low-cost post-training paradigm and narrows the gap between open and closed models.

DeepSeek R1 Explained

1. Overview

What is DeepSeek R1?

DeepSeek R1 is a reasoning-enhanced large language model released by DeepSeek. Through innovative post-training techniques, it achieves reasoning capabilities comparable to OpenAI o1 — while being completely free and open source.

Core Breakthroughs

| Breakthrough | Description | |------|------| | R1-Zero | First to show that pure reinforcement learning can produce emergent reasoning without supervised learning | | New training paradigm | RL-based post-training using the GRPO algorithm, significantly improving reasoning | | Distillation innovation | Distilling large-model capabilities into smaller models to bring reasoning to more scenarios |

Key Achievements

≈ o1 level: Reasoning performance on par with OpenAI o1
Completely free: A strong reasoning model available to everyone
Open source: Technical approach and methodology publicly released

---

2. The GRPO Algorithm

GRPO vs PPO

PPO (traditional) pipeline: 1. Input question → 2. Generate output → 3. Reward evaluation → 4. Value prediction → 5. Compute advantage → 6. Update policy

GRPO (R1's algorithm) pipeline: 1. Input question → 2. Generate multiple candidates in parallel → 3. Compare rewards → 4. Select the best → 5. Optimize directly

Performance Comparison

| Metric | PPO | GRPO | |------|-----|------| | Algorithm complexity | 85% | 45% | | Training efficiency | 60% | 90% | | Final performance | 70% | 85% |

Key Advantages of GRPO

Improved compute efficiency: Removing the Value Model greatly reduces memory and compute overhead
Suits open-ended problems: No reliance on a value model, better for complex reasoning tasks
Competitive optimization: Candidates compete against each other, continuously improving the model

---

3. Training Pipeline

Two-Stage Training Strategy

Stage 1: Reasoning capability training

| Step | Content | Data | |------|------|------| | 1 | SFT on a small amount of high-quality data | Thousands of chain-of-thought samples | | 2 | RL on reasoning-intensive tasks | Coding, math, science, and logic task sets |

Stage 2: General capability optimization

| Step | Content | Data | |------|------|------| | 1 | SFT on multi-domain data | 600K reasoning samples + 200K general samples | | 2 | RL for safety & general abilities | Comprehensive evaluation system |

---

4. Performance

Benchmark Comparison

| Benchmark | GPT-4 | OpenAI o1 | DeepSeek R1 | |----------|-------|-----------|-------------| | AIME 2024 | 83.3 | 93.0 | 94.2 | | Codeforces | 11 | 213 | 251 | | MATH-500 | 42.5 | 85.5 | 90.8 |

Capability Comparison

| Capability | DeepSeek R1 | OpenAI o1 | GPT-4 | |------|-------------|-----------|-------| | Math reasoning | 95 | 90 | 70 | | Coding | 92 | 88 | 75 | | Chinese understanding | 96 | 82 | 78 |

---

5. Technical Impact

Demonstrates the viability of pure reinforcement learning
Innovative application of the GRPO algorithm
Establishes a low-cost post-training paradigm
Performance parity between open-source and closed-source models

---

Source: Easy AI Tutorial Series