DeepSeek-R1 and GRPO: An Engineering Revolution in LLM Reinforcement Learning

Forum topic · 小凯 · 2026-05-22

Summary

This post explains GRPO (Group Relative Policy Optimization), the algorithm behind DeepSeek-R1 (arXiv:2501.12948), which removes the Critic model required by PPO. Instead of a value network, GRPO samples G answers per question, scores them, and uses the group's mean reward as a baseline: answers above average are reinforced, below-average ones suppressed. The author uses a Feynman-style archery analogy to show why relative in-group comparison supplies sufficient learning signal. Key findings covered include: pure RL (R1-Zero) can elicit strong math reasoning without SFT; models show sudden 'aha moment' jumps during RL training; cold-start SFT improves stability; and removing the Critic roughly halves GPU memory and compute costs. The post also raises critical questions: GRPO assumes reasonably accurate reward signals, which may break down in deep-research tasks lacking ground truth; and its popularity may partly reflect first-mover advantage and ecosystem lock-in rather than algorithmic optimality. It concludes that GRPO is a must-know baseline for LLM post-training engineers, while encouraging exploration of alternatives such as DAPO.

DeepSeek-R1 and GRPO: An Engineering Revolution in RL for LLM Reasoning

> Paper: DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948, 2025

The Core Problem

Training reasoning in LLMs traditionally requires two models: a "student" (Actor) that learns the policy, and a "teacher" (Critic) that evaluates quality. DeepSeek-R1 asked a bold question: what if you don't need the teacher?

GRPO (Group Relative Policy Optimization)'s surprising answer: no Critic needed — peer comparison within a group is enough.

What the Paper Actually Says

Traditional RL for LLMs uses PPO (Proximal Policy Optimization), which requires:

Actor: the policy model that generates answers
Critic: a value model estimating each state's value
The Critic is typically as large as the Actor, doubling training cost
The Critic's value estimates are extremely unstable on long sequences

GRPO's core innovation: drop the Critic and estimate the baseline using relative performance within a group.

Concretely:

1. For the same question, sample G answers from the current policy (a group) 2. Score each answer (reward) 3. Compute the group's mean score 4. Each answer's "advantage" = its own score − the mean 5. Reinforce the policy for above-average answers; suppress it for below-average ones

It's like being graded not against an absolute scale, but by your relative standing among classmates. You don't need to know "what a perfect score is" — only whether you're above or below the average.

A Feynman-Style Explanation: Why It Works

Imagine archery. PPO is like having a coach predict before each shot, "that arrow would score about X." The coach must be excellent — anticipating your form and the wind — and is expensive to train, and sometimes wrong.

GRPO is like firing 8 arrows in a burst, then seeing which landed close to the target and which didn't. For the close ones, note "that stance was right"; for the far ones, "that stance was wrong."

No score-predicting coach needed. You just need enough arrows for good and bad to differentiate naturally.

That's GRPO's insight: when the sample count is large enough, the relative differences within a group contain sufficient signal by themselves.

Key Findings

From the DeepSeek-R1 technical report:

1. Pure RL can elicit reasoning. DeepSeek-R1-Zero (pure RL, no SFT) reaches high performance on math reasoning. Reasoning isn't "taught" but "practiced" — the model discovers effective strategies through massive trial and error.

2. The Aha Moment. During RL training, the model suddenly "gets" certain reasoning tricks (e.g., "I should verify before concluding"). The training curve jumps abruptly rather than improving gradually. The report doesn't analyze the mechanism — is it genuine capability emergence or a training-noise artifact? Still unclear.

3. Cold-start SFT improves stability. DeepSeek-R1 (with cold-start SFT) is more stable than R1-Zero. A small amount of high-quality SFT data helps the model converge faster in the RL stage, reducing random early exploration.

4. GRPO significantly cuts training cost. Removing the Critic reduces GPU memory and compute time by roughly half, making reasoning-model training feasible in resource-constrained settings.

The Deeper Insight

GRPO's greatest value isn't the engineering trick of removing the Critic, but what it reveals about how LLMs learn:

An LLM doesn't need a perfect evaluator to learn reasoning. It only needs enough opportunities to try and fail, plus a relatively objective scoring standard.

This mirrors human learning. When learning to ride a bike, you don't need a coach constantly telling you "your balance score is 7.3/10." You ride, fall a few times, feel "that posture kept me up," and adjust.

But GRPO has a fundamental limitation: it assumes reward signals are accessible and reasonably accurate. If the reward itself is noisy (e.g., a frequently wrong verifier), the group-relative baseline absorbs that noise too.

In Deep Research scenarios this is especially severe: final answers often have no clear ground truth (e.g., "analyze the US–China semiconductor competition"), so the reward signal itself is fuzzy. Does GRPO still work there? That's the biggest open question.

A Critical Perspective

A Feynman-style challenge: "Did GRPO succeed because it captures the essence of reasoning learning, or because it happens to fit current LLM training constraints?"

Three reasons for GRPO's popularity:

1. No Critic — simpler engineering 2. It comes from DeepSeek, whose models are strong, so others follow 3. It demonstrably works on math and code

But none of these prove GRPO is "the essence." Other algorithms (e.g., DAPO, Mu-GRPO) may do better in some settings; GRPO's popularity may partly reflect first-mover advantage and ecosystem lock-in. The technical report itself is honest — it never claims GRPO is optimal. But the community sometimes equates "popular" with "best." That's cargo cult thinking.

Conclusion

DeepSeek-R1 and GRPO are milestones in LLM post-training. They demonstrated that:

1. Pure RL can train strong reasoning ability 2. RL without a Critic is viable 3. Reasoning can be acquired through trial-and-error plus relative scoring

Open questions remain:

Does GRPO still work on tasks without ground truth?
Do group size and sampling strategy need per-task tuning?
Beyond GRPO, which other "Critic-free" RL algorithms deserve exploration?

For engineers: if you're doing LLM post-training, GRPO is a must-know baseline — but not the only choice.

For researchers: GRPO's success points toward "simplifying RL architectures." More concise, more effective RL algorithms may follow.

> "The first principle is that you must not fool yourself." GRPO is simple and elegant, but don't believe it's perfect just because it's simple. There are no perfect algorithms — only algorithms suited to specific problems.