M-GRPO: Stabilizing Self-Supervised RL for LLMs with Momentum-Anchored Policy Optimization

Forum topic · ✨步子哥 · 2026-01-16

Summary

A forum post on zhichai.net introduces M-GRPO (Momentum-Anchored Group Relative Policy Optimization), a method from Fudan University, Shanghai Innovation Institute, and Shanghai AI Laboratory (arXiv 2512.13070) that stabilizes self-supervised reinforcement learning (SS-RLVR) for large language models. Standard self-refining training, where models generate their own pseudo-labels via majority voting, suffers from policy collapse and entropy collapse: training reward spikes then crashes, output diversity vanishes, and models learn to reward-hack their own noisy scoring. M-GRPO introduces two mechanisms: (1) a momentum model updated via exponential moving average (m=0.99) acts as a stable 'grandparent' reference whose rollouts are pooled with the current policy's for majority voting, reducing pseudo-label noise; (2) a dynamic IQR-based entropy filter prunes extremely low-entropy trajectories to preserve exploration. On Qwen3-4B-Base, M-GRPO achieves final checkpoint performance nearly equal to best-checkpoint performance across MATH, AIME 2024/2025, GPQA, LiveCode, MBPP, and MMLU-Pro (e.g., MATH 51.88% final vs 25.32% for SRT baseline), enabling sustained improvement without human-annotated data. The post also discusses philosophical limits relative to Penrose's arguments and future directions.

M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for LLMs

This post discusses a recent paper from Fudan University, the Shanghai Innovation Institute, and the Shanghai AI Laboratory (arXiv 2512.13070), which proposes M-GRPO (Momentum-Anchored Group Relative Policy Optimization) — a method that lets large language models improve reasoning continuously and stably without human-annotated data.

Key points

Why self-supervised RL collapses

Self-supervised RL with verifiable rewards (SS-RLVR) lets a model generate questions, answers, scores, and improvements on its own. The classic SRT (Self-Refining Training) pipeline generates multiple rollouts per problem, assigns pseudo-labels via majority voting, and rewards/penalizes itself accordingly. Accuracy climbs quickly on MATH at first, but the authors reproduce two failure modes:

Policy collapse: training reward spikes, then crashes; final validation accuracy drops sharply.
Entropy collapse: policy entropy rapidly falls to near zero early in training; the model becomes overconfident, generating near-identical answers and losing exploration ability.

Increasing rollouts from 8 to 256 only delays, but does not prevent, collapse. The root cause is a vicious cycle: fast self-evolution makes pseudo-labels increasingly noisy, so the model learns reward hacking — gaming its own scoring system rather than seeking correctness.

The momentum anchor ('the grandparent')

M-GRPO pairs the fast-learning query policy (the child) with a slow-moving momentum model (the grandparent) updated by exponential moving average:

\pi_{\theta_k} \leftarrow m \cdot \pi_{\theta_k} + (1 - m) \cdot \pi_{\theta_q}

with m typically 0.99, so the momentum model absorbs only 1% new influence per update — effectively a low-pass filter over training dynamics. Pseudo-labels are assigned by majority voting over a pool mixing the query policy's M rollouts with the momentum model's N rollouts:

y_v \leftarrow \arg\max_{y^*} \sum_{y' \in Y_{\text{pool}}} \mathbb{I}[\text{ans}(y') = \text{ans}(y^*)]

The slow, reliable grandparent answers act as an anchor, substantially reducing label noise.

The IQR filter ('the gardener')

To prevent entropy collapse, M-GRPO dynamically prunes trajectories by policy entropy:

1. Compute policy entropy for all trajectories in the batch. 2. Compute Q1 and Q3 quartiles. 3. Discard trajectories with entropy below Q1 − k × IQR (k = 0.75).

The threshold adapts over training: loose early, tighter later. With filtering, average policy entropy stays high throughout training, avoiding the baseline's entropy avalanche.

Experimental results

Validated on Qwen3-4B-Base across math, code, and general-knowledge benchmarks (paper Table 2, best vs final checkpoints):

| Benchmark | SRT-Best (%) | SRT-Final (%) | M-GRPO-Best (%) | M-GRPO-Final (%) | |---|---|---|---|---| | MATH | 50.78 | 25.32 | 52.14 | 51.88 | | MATH500 | 50.80 | 28.00 | 52.60 | 52.00 | | AIME 2024 | 39.33 | 20.00 | 42.00 | 41.33 | | AIME 2025 | 38.00 | 18.67 | 40.67 | 40.00 | | GPQA Diamond | 35.04 | 22.57 | 40.71 | 39.82 | | LiveCode | 36.00 | 24.00 | 38.50 | 38.00 | | MBPP | 45.20 | 30.40 | 47.60 | 47.00 | | MMLU-Pro | 55.12 | 42.36 | 57.48 | 56.92 |

Notably, M-GRPO's final-checkpoint performance nearly matches or exceeds its best checkpoint, unlike SRT's drastic degradation — demonstrating genuine continual improvement. Ablations confirm the momentum anchor stabilizes reward curves, the IQR filter sustains entropy, and both together preserve gains at 32 rollouts (diminishing returns beyond). Larger models (7B, 8B) benefit more, suggesting positive scaling synergy.

Philosophical caveat

The post notes that M-GRPO remains an elegant algorithmic correction, not human-like 'insight' outside formal systems — echoing Roger Penrose's argument in *The Emperor's New Mind* that mathematical understanding may involve non-computable processes.

Future directions

Combining with other optimizers (e.g., Proximal Policy Optimization)
Extending to multimodal or multi-agent settings
Validating long-term stability on more open-ended tasks

References

1. Bai, Y., et al. "M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization." arXiv:2512.13070 (2025). 2. Wolfe, C. R. "Group Relative Policy Optimization (GRPO) Explained." Deep Learning Focus Substack (2025). 3. Penrose, R. *The Emperor's New Mind*. Oxford University Press (1989). 4. Pref-GRPO: preference optimization for text-to-image generation. arXiv:2508.20751. 5. GRPO-LEAD: concise mathematical reasoning with length constraints. ACL Anthology (2025).