VL-Rethinker: How Selective Sample Replay and Forced Rethinking Fix Vanishing Advantages in VLM Reinforcement Learning

Forum topic · 小凯 · 2026-05-11

Summary

VL-Rethinker (arXiv:2504.08837), from HKUST and University of Waterloo researchers, addresses why GRPO-based reinforcement learning that enabled long chain-of-thought reasoning in text LLMs like DeepSeek-R1 fails on vision-language models. The authors identify a core problem: vanishing advantages — when all sampled responses receive identical binary rewards, GRPO advantages become 0/0 and gradient signal disappears, leaving over 80% of training data ineffective late in training. Two techniques solve this: Selective Sample Replay (SSR), a prioritized replay buffer that reuses samples with non-zero advantages as dynamic curriculum learning, and Forced Rethinking, which appends rethinking triggers during rollout, applies SFT loss only to successful reflection trajectories, and induces spontaneous self-reflection at inference time without prompts. Using pure RL on 38,870 queries, VL-Rethinker-72B reaches 80.4% on MathVista and 63.5% on MathVerse, surpassing OpenAI o1, while the 7B variant outperforms OpenVLThinker and MM-Eureka. The paper also reports an emergent metacognitive behavior: the model learns to question flawed problem premises.

VLMs Don't Learn to Think on Their Own — Unless You Whip Them: The Brutal Truth of VL-Rethinker 🎯

> Core thesis: Vision-language models (VLMs) and text LLMs are fundamentally different creatures under reinforcement learning. DeepSeek-R1 learns long chain-of-thought with plain GRPO, but Qwen2.5-VL with the same method stalls. Teams from HKUST and Waterloo identified a problem everyone overlooked: vanishing advantages — once the model gets strong enough, each sample group is either all-correct or all-wrong, and GRPO gradients die. Their solution: first recover the lost gradients with SSR, then force reflection with Forced Rethinking.

---

1. A Perplexing Observation 🤔

From late 2024 to early 2025, the slow-thinking revolution swept through text LLMs:

| Model | Training Method | Long CoT Emerges Naturally | Math Gains | |:---:|:---:|:---:|:---:| | DeepSeek-R1 | GRPO (pure RL) | ✅ Yes | +30%+ | | Kimi-1.5 | RL + long context | ✅ Yes | Significant | | QwQ | RL | ✅ Yes | Significant | | Qwen2.5-VL-72B | GRPO (pure RL) | ❌ No | Marginal |

> What's wrong? With the same GRPO, why do text models learn reflection while VLMs don't?

The HKUST/Waterloo answer: it's not GRPO's fault — VLM GRPO suffers a specific late-training collapse called vanishing advantages.

---

2. Vanishing Advantages: When the Model Gets Too Strong, the Gradients Die 💀

GRPO samples $$G$$ responses per query and normalizes group rewards into advantages:

\hat{A}_{i,t} = \frac{r(x, y_i) - \text{mean}(\{r(x, y_1), \dots, r(x, y_G)\})}{\text{std}(\{r(x, y_1), \dots, r(x, y_G)\})}

> Fatal weakness: if all samples are correct (reward=1) or all wrong (reward=0), both numerator and standard deviation are zero — the advantage becomes $$0/0$$ and the gradient signal vanishes.

On text LLMs this is less severe because math problem difficulty is spread out and early-training error rates are high. But tracking Qwen2.5-VL-72B's training dynamics:

| Training Stage | Effective queries (non-zero advantage) | |:---:|:---:| | Initial | ~40% | | After 16×16 gradient steps | < 20% |

> 80% of training data becomes noise. The model is "learning" — from thin air.

Consequences: shrinking effective batch size, training instability, and premature convergence to "safe" pattern-matching instead of deeper reasoning.

---

3. SSR: Rescuing Gold from the Garbage Heap of History 🏆

Selective Sample Replay (SSR) is blunt and simple: since many current-batch samples carry no gradient, retrieve "gradient-bearing" samples from past training.

SSR maintains a replay buffer $\mathcal{B}_{\text{replay}}$ storing only samples with $|\hat{A}_k| > 0$ , then samples by advantage magnitude:

P(\text{select } j) = \frac{|\hat{A}_j|^{\alpha}}{\sum_{k \in \mathcal{B}_{\text{replay}}} |\hat{A}_k|^{\alpha}}

> Replaying high-|advantage| samples — large positives or large negatives — targets the model's decision boundary, acting as dynamic online curriculum learning.

Ablations:

| Method | MathVision | MathVista | MathVerse | |:---:|:---:|:---:|:---:| | Pure GRPO | 26.0% | 70.9% | 51.4% | | GRPO + Filter (no SSR) | 28.5% | 72.0% | 50.0% | | GRPO + SSR | 32.3% | 74.9% | 54.2% |

> SSR alone contributes 3.8 points on MathVision — at 7B scale, the gap from "mediocre" to "leading."

---

4. Forced Rethinking: If the Model Won't Reflect, Make It 🪓

Even with stable training, VLMs don't spontaneously produce reflection. Text LLMs develop "Wait", "Alternatively" patterns under GRPO; VLMs almost never do. One hypothesis: multimodal reward signals are too noisy to build a clear "reflection → better outcome" causal chain.

Mechanism

1. Model generates initial answer $$y_1$$ 2. Append a rethinking trigger (e.g., "recheck your reasoning") 3. Model continues generating $$y_2$$ (reflection/correction) 4. Full sequence: $y = y_1 \oplus \text{trigger} \oplus y_2$ 5. Apply only to a fraction ( $$q < 1$$ ) of samples 6. Keep only trajectories whose final answers are correct 7. Apply extra SFT loss on those successful trajectories

> Key design: at inference the model needs no forced trigger — it learned to reflect spontaneously when needed. Forced Rethinking is a training crutch, not an inference-time shackle.

Three trigger types

| Trigger Type | Function | Example | |:---|:---|:---| | Self-verification | Verify reasoning steps | "Verify each step of the above reasoning" | | Self-correction | Fix errors | "Check whether the above answer is wrong" | | Self-questioning | Challenge assumptions | "Question the premises of the above reasoning" |

| Configuration | MathVision | MathVista | MathVerse | |:---:|:---:|:---:|:---:| | SSR only | 29.8% | 72.4% | 53.2% | | SSR + Forced Rethinking | 32.3% | 74.9% | 54.2% |

> Forced Rethinking adds 2.5 points on MathVision — and more importantly, teaches the model reflection as a capability, not just a score.

---

5. Results: The VLM Finally Thinks Slowly 🚀

72B vs the World

| Benchmark | OpenAI o1 | Qwen2.5-VL-72B | VL-Rethinker-72B | Gap | |:---:|:---:|:---:|:---:|:---:| | MathVista | 73.9% | 74.8% | 80.4% | +6.6% vs o1 | | MathVerse | 57.0% | 57.2% | 63.5% | +6.5% vs o1 | | MathVision | 60.3% | 38.1% | 44.9% | -15.4% vs o1 | | MMMU-Pro | 62.4% | 51.6% | 55.9% | -6.5% vs o1 | | EMMA | 45.7% | 34.1% | 38.5% | +4.4% vs o1 | | MEGA | 56.2% | 49.0% | 51.3% | +2.3% vs o1 |

> On math reasoning, VL-Rethinker-72B beats OpenAI o1 — no distillation, no copying o1's CoT, pure RL from scratch.

7B vs Open-Source Peers

| Benchmark | OpenVLThinker-7B | MM-Eureka-7B | VL-Rethinker-7B | |:---:|:---:|:---:|:---:| | MathVista | 70.2% | 73.0% | 74.9% | | MathVerse | 47.9% | 50.3% | 54.2% | | MathVision | 25.3% | 26.9% | 32.3% | | MMMU-Pro | 37.3% | — | 41.7% |

> At 7B, VL-Rethinker sweeps open-source multimodal reasoning models using only pure RL + two clever tricks.

---

6. My Bet 💰

I bet $1,000: by end of 2025, every mainstream VLM RL pipeline will integrate SSR-style experience replay and Forced-Rethinking-style reflection induction.

Why:

1. Vanishing advantages are structural: binary rewards + GRPO + strong models guarantees the problem. SSR is a necessity, not a nicety. 2. VLM reflection doesn't emerge naturally — unlike text LLMs. Forced Rethinking shows external guidance can be internalized. 3. The pure-RL route works: no o1 distillation, no long-CoT SFT data — just 38K queries of pure RL beat o1 on MathVista and MathVerse. 4. Implementation cost is tiny: SSR is a few lines of replay code; Forced Rethinking is one appended sentence. Training tricks spread far faster than architecture innovations.

Who are the enemies? Teams still distilling o1/R1 CoTs at 10× the data for 50% of the effect; RL dogmatists who believe "GRPO is universal" (GRPO dies on VLMs — SSR is its CPR); researchers who ignore multimodal specificity.

---

7. The Most Striking Finding: The Model Learned to Question the Question Itself 🤯

In the paper's Figure 3, VL-Rethinker, while reflecting, discovers a flaw in the problem itself — recognizing contradictory given conditions rather than blindly solving an unsolvable problem.

This was never explicitly taught. It's an emergent metacognitive ability — the model learned to check not only its answers, but the validity of the question. All current benchmarks assume problems are correct; real-world problems often aren't. An AI that questions the problem is closer to genuine intelligence than one that only solves it.

---

Paper Details

| Item | Content | |:---|:---| | Title | VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning | | Authors | Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen | | Institutions | HKUST, University of Waterloo, INF.AI, Vector Institute | | arXiv ID | 2504.08837 | | Date | 2025-04-10 | | Core contributions | Selective Sample Replay (SSR) for vanishing advantages; Forced Rethinking to induce VLM self-reflection | | Training data | 38,870 queries (16K/20K after cleaning) | | Best results | MathVista 80.4% (72B), MathVerse 63.5% (72B) — surpassing OpenAI o1 |