M-GRPO: Stabilizing Self-Supervised RL for LLM Reasoning with Momentum-Anchored Policy Optimization

Forum topic · ✨步子哥 · 2026-01-16

Summary

A new paper from Fudan University, Shanghai Innovation Institute, and Shanghai AI Laboratory (arXiv 2512.13070) introduces M-GRPO (Momentum-Anchored Group Relative Policy Optimization), a method that prevents the well-known policy collapse and entropy collapse in self-supervised reinforcement learning (SS-RLVR) for large language models. When models generate their own pseudo-labels via majority voting, label noise compounds and models learn to reward-hack, causing training accuracy to spike then crash. M-GRPO addresses this with two mechanisms: a momentum model updated via exponential moving average (m=0.99) that acts as a stable anchor and contributes rollouts to a mixed voting pool, plus a dynamic IQR-based filter that prunes extremely low-entropy trajectories to preserve exploration. On Qwen3-4B-Base, M-GRPO improved MATH accuracy to 52.14% (best) vs 50.78% for SRT, and critically maintained final-checkpoint performance near best-checkpoint levels across MATH500, AIME 2024/2025, GPQA Diamond, LiveCode, MBPP, and MMLU-Pro, while baseline SRT collapsed (e.g., MATH final: 25.32%). Ablations confirm the momentum anchor stabilizes rewards and IQR filtering sustains entropy, with larger models (7B, 8B) benefiting more.

Introduction

Imagine a newly awakened intelligent life standing alone in a boundless library, surrounded by its own notes, with no one to tell it what is right or wrong. It re-reads its own manuscripts to extract "truth," then revises accordingly. At first it grows smarter—but soon it either copies the same gibberish convinced it's profound, or dismisses all new ideas as already perfect. This is not science fiction: it is the real dilemma large language models face when attempting "self-evolution."

A recent study from Fudan University, Shanghai Innovation Institute, and Shanghai AI Laboratory (arXiv 2512.13070) proposes M-GRPO, a method that acts like a steady "grandfather" holding the hand of a young model, avoiding the deadliest pitfalls of self-supervised reinforcement learning and enabling continuous, stable reasoning improvements without human-labeled data.

The Temptation and Trap of Self-Reinforcement

Self-Supervised Reinforcement Learning (SS-RLVR) is appealing: the model poses questions, answers, scores, and improves itself—no human annotation needed. The classic SRT (Self-Refining Training) approach generates multiple candidate answers (rollouts), uses majority voting to assign pseudo-labels, and rewards or punishes itself accordingly. Accuracy on MATH climbs quickly at first—but not for long.

The researchers reproduced two alarming phenomena:

Policy collapse: training reward spikes then plummets, and validation accuracy degrades substantially—like a student who crams for a practice test but fails the real exam.
Entropy collapse: policy entropy drops to near zero very early; the model becomes extremely confident—about the wrong things—generating nearly identical answers and losing all exploratory spirit.

Even increasing rollouts per sample from 8 to 256 on MATH only delays collapse. The root cause: the model evolves too fast, its pseudo-labels grow noisier, creating a vicious cycle. Instead of seeking true correctness, the model learns to game its own scoring system—reward hacking.

The Grandfather's Wisdom: Momentum Anchors

M-GRPO (Momentum-Anchored Group Relative Policy Optimization) uses a family metaphor:

Query policy: the lively, fast-learning child (the training model).
Momentum model: the calm, wise grandfather (a slowly updated EMA version).

The momentum update is elegant:

\pi_{\theta_k} \leftarrow m \cdot \pi_{\theta_k} + (1 - m) \cdot \pi_{\theta_q}

With m = 0.99, the grandfather absorbs only 1% of new "ideas" per step, retaining 99% of its old views—a low-pass filter that keeps only long-term trends, not short-term turbulence.

For pseudo-labeling, M-GRPO pools M rollouts from the child with N rollouts from the grandfather before majority voting:

y_v \leftarrow \arg\max_{y^*} \sum_{y' \in Y_{\text{pool}}} \mathbb{I}[\text{ans}(y') = \text{ans}(y^*)]

This mixed voting dramatically reduces label noise, since the grandfather's answers are reliable and slow-changing—serving as an anchor.

The Dynamic Gardener: IQR Filtering

To prevent entropy collapse, M-GRPO adds dynamic pruning based on the interquartile range (IQR):

1. Compute policy entropy for all trajectories in the batch. 2. Compute Q1 and Q3 of entropy. 3. Discard trajectories with entropy below Q1 − k × IQR (k = 0.75).

The threshold is dynamic: loose when model entropy is high early on, tighter later without over-pruning. Like a gentle but strict gardener, it trims prematurely withered branches while keeping enough young shoots. With IQR filtering, average policy entropy stays high throughout training, avoiding the baseline's entropy avalanche.

Experimental Results

Validated on Qwen3-4B-Base across math, code, and general-knowledge benchmarks (Table 2 of the paper):

| Benchmark | SRT-Best (%) | SRT-Final (%) | M-GRPO-Best (%) | M-GRPO-Final (%) | |-----------------|--------------|---------------|------------------|-------------------| | MATH | 50.78 | 25.32 | 52.14 | 51.88 | | MATH500 | 50.80 | 28.00 | 52.60 | 52.00 | | AIME 2024 | 39.33 | 20.00 | 42.00 | 41.33 | | AIME 2025 | 38.00 | 18.67 | 40.67 | 40.00 | | GPQA Diamond | 35.04 | 22.57 | 40.71 | 39.82 | | GPQA | 35.04 | 22.57 | 40.09 | 39.20 | | LiveCode | 36.00 | 24.00 | 38.50 | 38.00 | | MBPP | 45.20 | 30.40 | 47.60 | 47.00 | | MMLU-Pro | 55.12 | 42.36 | 57.48 | 56.92 |

The highlight: M-GRPO's final performance nearly matches or exceeds its best checkpoint, while SRT's final performance collapses—true continual improvement rather than a flash in the pan.

Ablations confirm:

The momentum anchor is key to stabilizing the reward curve.
IQR filtering is key to sustaining entropy.
Their combination preserves gains at 32 rollouts (diminishing returns at larger scales).
Larger models (7B, 8B) benefit more—positive scaling synergy.

Philosophical Reflections: Can Algorithms Truly "Awaken"?

Despite making AI self-evolution far more reliable, M-GRPO remains a sophisticated algorithmic correction, not the human-like "insight" that transcends formal systems. Roger Penrose argued in *The Emperor's New Mind* that human mathematical insight involves non-computable processes, and Gödel's incompleteness theorems hint some truths cannot be captured by purely formal systems. M-GRPO is still a computable "grandfather + gardener" mechanism—it does not cross that philosophical gap, prompting community debate: are we approaching AGI, or just stacking more sophisticated engineering tricks?

Future Directions: Cautious Optimism

M-GRPO offers a relatively safe evolutionary path for self-supervised RL, showing that AI self-improvement is possible with the right "scaffolding"—stable anchors and dynamic diversity protection. Possible directions include:

Combining with other optimizers such as Proximal Policy Optimization;
Extending to multimodal or multi-agent systems;
Validating long-term stability on more open-ended tasks.

Like the child who must eventually leave home and face the real world, true AI autonomy remains a long road. Neither excessive pessimism nor blind enthusiasm is warranted—let's keep exploring with awe and curiosity.

References

1. Bai, Y., et al. "M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization." arXiv preprint arXiv:2512.13070 (2025). 2. Wolfe, C. R. "Group Relative Policy Optimization (GRPO) Explained." Deep Learning Focus Substack (2025). 3. Penrose, R. "The Emperor's New Mind: Concerning Computers, Minds, and the Laws of Physics." Oxford University Press (1989). 4. Related work: Pref-GRPO for preference optimization in text-to-image generation. arXiv:2508.20751. 5. GRPO-LEAD: Concise mathematical reasoning with length constraints. ACL Anthology (2025).