DeepSeek-R1: The Engineering Revolution of the GRPO Algorithm

> Paper: DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948, 2025

The Core Question

Training LLM reasoning traditionally required two models: a "student" (Actor) learning the policy and a "teacher" (Critic) evaluating quality. DeepSeek-R1 asked a bold question: what if no teacher were needed?

GRPO's (Group Relative Policy Optimization) surprising answer: no Critic needed—peer comparison is enough.

What the Paper Actually Says

Traditional RL for LLMs uses PPO (Proximal Policy Optimization), which requires:

• Actor: the policy model generating answers
• Critic: a value-function model evaluating each state
• The Critic is typically the same size as the Actor, doubling training cost
• Critic value estimates are extremely unstable over long sequences

GRPO's core innovation: drop the Critic and estimate the baseline from relative performance within a group.

The mechanics: 1. For the same question, sample G answers from the current policy (a group) 2. Score each answer (reward) 3. Compute the group's average score 4. Each answer's "advantage" = its own score − group average 5. Reinforce the policy for above-average answers; weaken it for below-average ones

It's like an exam without absolute rankings—what matters is your relative standing among classmates. You never need to know "what a perfect score is," only whether you're above or below average.

The Feynman View: Why Does It Work?

Imagine archery. PPO is like having a coach who, before each shot, predicts "this arrow is worth about X points." That coach must be extremely skilled—anticipating your form and the wind—and expensive to train, and even then sometimes wrong.

GRPO is like firing 8 arrows in one volley, then seeing which landed near the bullseye. For the close ones: "that stance was right." For the far ones: "that stance was off."

No score-predicting coach needed. Just enough arrows that "good" and "bad" naturally separate.

That's GRPO's insight: when sampling is sufficient, the relative differences within a group contain enough signal by themselves.

Key Findings

1. Pure RL can elicit reasoning. DeepSeek-R1-Zero (pure RL, no SFT) reaches high levels on math reasoning. Reasoning isn't "taught"—it's "practiced": through massive trial and error, the model discovers effective reasoning strategies on its own.

2. The Aha Moment. During RL training, the model suddenly "gets" a reasoning technique (e.g., "I should verify before concluding"). These jumps are abrupt, not gradual. Is it genuine capability emergence or a training-noise artifact? The report doesn't dig into the mechanism; it remains unclear.

3. Cold-start SFT improves stability. DeepSeek-R1 (with cold-start SFT) is more stable than R1-Zero. A small amount of high-quality SFT data helps the model converge faster during RL and reduces random early exploration.

4. GRPO significantly cuts training cost. Removing the Critic cuts GPU memory and compute time roughly in half, making reasoning-model training feasible in resource-constrained settings.

The Real Insight

GRPO's greatest value isn't the engineering trick of "removing the Critic"—it's what it reveals about how LLMs learn:

An LLM doesn't need a perfect evaluator to learn reasoning. It only needs enough opportunities to try, plus a relatively objective judging criterion.

This matches human learning. You don't learn to ride a bike with a coach constantly announcing "your balance score is 7.3/10." You ride, you fall, you notice "that posture kept me up," and you adjust.

GRPO formalizes this intuition. Its simplicity is itself a kind of beauty.

But GRPO has a fundamental limitation: it assumes the reward signal is accessible and reasonably accurate. If rewards are noisy (e.g., the verifier is often wrong), the group-relative baseline learns that noise too.

In Deep Research settings this is especially severe—final answers often have no clear ground truth (e.g., "analyze US-China semiconductor competition"), so the reward itself is ambiguous. Does GRPO still work there? That's the biggest open question.

A Critical Perspective

A Feynman-style challenge: "Is GRPO's success because it captures the essence of reasoning learning, or because it happens to fit current LLM training constraints?"

Three reasons for GRPO's popularity: 1. No Critic needed—simpler engineering 2. It comes from DeepSeek, whose models are strong, so people follow 3. It genuinely works on math and code tasks

But none of these prove GRPO is "the essence." Other algorithms (DAPO, Mu-GRPO, etc.) may work better in some settings. Part of GRPO's popularity may be first-mover advantage and ecosystem lock-in.

The technical report itself is honest—it never claims GRPO is "optimal." But the community sometimes equates "popular" with "best." That's cargo cult.

Conclusion

DeepSeek-R1 and GRPO are milestones in LLM post-training. They demonstrate: 1. Pure RL can train powerful reasoning capability 2. Critic-free RL is viable 3. Reasoning can be acquired through "trial-and-error + relative judgment"

But open questions remain:

• Does GRPO still work on tasks without ground truth?
• Do group size and sampling strategy need per-task tuning?
• Beyond GRPO, what other "Critic-free" RL algorithms deserve exploration?

For engineers: if you're doing LLM post-training, GRPO is a must-know baseline—but not the only option.

For researchers: GRPO's success highlights the direction of "simplifying RL architectures." More "simpler, more effective" RL algorithms may be coming.

> "The first principle is that you must not fool yourself." GRPO is elegant—but don't trust it just because it's simple. There are no perfect algorithms, only algorithms suited to specific problems.