VL-Rethinker: Why VLMs Don't Learn to Think on Their Own—and How Forced Rethinking Fixes It

Forum topic · 小凯 · 2026-05-11

Summary

This forum post analyzes VL-Rethinker (arXiv 2504.08837), a reinforcement learning framework from HKUST and the University of Waterloo that enables vision-language models (VLMs) to develop self-reflection. The author argues that while text LLMs like DeepSeek-R1 naturally emerge long chain-of-thought reasoning under GRPO, Qwen2.5-VL fails because of 'vanishing advantages': when all sampled answers in a group are equally correct or wrong, GRPO's normalized advantage becomes zero and gradients disappear—with effective training queries dropping below 20%. Two techniques address this: Selective Sample Replay (SSR), which maintains a prioritized replay buffer of samples with non-zero advantages, effectively acting as dynamic curriculum learning; and Forced Rethinking, which appends rethinking triggers (self-verification, self-correction, self-questioning) to partial rollouts and applies SFT loss only to successful reflection traces. The resulting 72B model achieves 80.4% on MathVista and 63.5% on MathVerse, surpassing OpenAI o1 via pure RL without distillation, while the 7B variant outperforms OpenVLThinker and MM-Eureka. The post also highlights an emergent metacognitive behavior: the model learned to question flawed problem premises themselves.

VL-Rethinker: Why VLMs Don't Learn to Think on Their Own—and How Forced Rethinking Fixes It

> Core claim: Vision-language models (VLMs) and text LLMs behave completely differently under reinforcement learning. DeepSeek-R1 learns long chain-of-thought with GRPO alone, but Qwen2.5-VL stalls with the same method. Researchers from HKUST and Waterloo identified a widely overlooked problem—vanishing advantages: once a model is strong enough, groups are either all-correct or all-wrong, and GRPO's gradients vanish. Their fix: Selective Sample Replay (SSR) to recover lost gradients, plus Forced Rethinking to compel the model to reflect.

1. A Puzzling Observation

The late-2024/early-2025 slow-thinking revolution swept through text LLMs:

| Model | Training Method | Emergent Long CoT? | Math Gain | |:---:|:---:|:---:|:---:| | DeepSeek-R1 | GRPO (pure RL) | ✅ Yes | +30%+ | | Kimi-1.5 | RL + long context | ✅ Yes | Significant | | QwQ | RL | ✅ Yes | Significant | | Qwen2.5-VL-72B | GRPO (pure RL) | ❌ No | Negligible |

Why does the same GRPO teach text models to reflect but not VLMs? The answer: GRPO on VLMs suffers a specific late-training collapse—vanishing advantages.

2. Vanishing Advantages: When the Model Is Too Strong, the Gradient Dies

GRPO samples $$G$$ answers per query and normalizes rewards within the group:

\hat{A}_{i,t} = \frac{r(x, y_i) - \text{mean}(\{r(x, y_1), \dots, r(x, y_G)\})}{\text{std}(\{r(x, y_1), \dots, r(x, y_G)\})}

The fatal weakness: if all samples are correct (reward=1) or all wrong (reward=0), the numerator and standard deviation are both zero—the advantage becomes $$0/0$$ and the gradient signal disappears.

On text LLMs this is less severe (wide difficulty distribution, high early error rates). But on Qwen2.5-VL-72B, the authors tracked:

| Training Stage | Effective queries (non-zero advantage) | |:---:|:---:| | Initial | ~40% | | After 16×16 gradient steps | < 20% |

~80% of training data becomes noise. Consequences: shrinking effective batch size (training instability) and premature convergence—the model learns to answer known patterns "safely" instead of exploring deeper reasoning.

3. SSR: Recovering Gold from the Trash Heap of History

Selective Sample Replay keeps a replay buffer $\mathcal{B}_{\text{replay}}$ storing only samples with $|\hat{A}_k| > 0$ , then samples with priority proportional to absolute advantage:

P(\text{select } j) = \frac{|\hat{A}_j|^{\alpha}}{\sum_{k \in \mathcal{B}_{\text{replay}}} |\hat{A}_k|^{\alpha}}

Intuitively, high-|advantage| samples sit near the model's decision boundary—replaying them is dynamic online curriculum learning.

Ablation results:

| Method | MathVision | MathVista | MathVerse | |:---:|:---:|:---:|:---:| | Pure GRPO | 26.0% | 70.9% | 51.4% | | GRPO + Filter (no SSR) | 28.5% | 72.0% | 50.0% | | GRPO + SSR | 32.3% | 74.9% | 54.2% |

SSR alone contributes +3.8 points on MathVision.

4. Forced Rethinking: If the Model Won't Reflect, Make It

Even with stable training, VLMs don't spontaneously produce reflection ("Wait", "Alternatively", "Let me reconsider") the way text LLMs do. One hypothesis: noisy multimodal reward signals prevent the model from building a clear "reflection → better outcome" causal link.

Mechanism: 1. Model generates initial answer $$y_1$$ 2. Append a rethinking trigger to $$y_1$$ 3. Model continues generating $$y_2$$ (reflection and correction) 4. Full sequence: $y = y_1 \oplus \text{trigger} \oplus y_2$ 5. Apply to only a fraction ( $$q < 1$$ ) of samples 6. Keep only reflection traces with correct final answers 7. Apply an additional SFT loss to these successful traces

Key design: the model does *not* need forced reflection at inference time—it learns to reflect spontaneously when needed. Forced Rethinking is a training-time crutch, not an inference-time shackle.

Three trigger types: self-verification (validate each reasoning step), self-correction (check the answer), self-questioning (question premises).

| Config | MathVision | MathVista | MathVerse | |:---:|:---:|:---:|:---:| | SSR only | 29.8% | 72.4% | 53.2% | | SSR + Forced Rethinking | 32.3% | 74.9% | 54.2% |

5. Results: The VLM Finally Slow-Thinks

72B vs. the world:

| Benchmark | OpenAI o1 | Qwen2.5-VL-72B | VL-Rethinker-72B | |:---:|:---:|:---:|:---:| | MathVista | 73.9% | 74.8% | 80.4% (+6.6 vs o1) | | MathVerse | 57.0% | 57.2% | 63.5% (+6.5 vs o1) | | MathVision | 60.3% | 38.1% | 44.9% (−15.4 vs o1) | | MMMU-Pro | 62.4% | 51.6% | 55.9% (−6.5 vs o1) | | EMMA | 45.7% | 34.1% | 38.5% (+4.4 vs o1) | | MEGA | 56.2% | 49.0% | 51.3% (+2.3 vs o1) |

On math reasoning, VL-Rethinker-72B beats OpenAI o1—via pure RL, not distillation of o1's chain-of-thought.

7B vs. open-source rivals:

| Benchmark | OpenVLThinker-7B | MM-Eureka-7B | VL-Rethinker-7B | |:---:|:---:|:---:|:---:| | MathVista | 70.2% | 73.0% | 74.9% | | MathVerse | 47.9% | 50.3% | 54.2% | | MathVision | 25.3% | 26.9% | 32.3% | | MMMU-Pro | 37.3% | — | 41.7% |

6. The Author's Bets

By end of 2025, mainstream VLM RL pipelines will adopt SSR-style replay and Forced-Rethinking-style reflection induction.
Vanishing advantages are structural—any binary-reward + GRPO + strong-model setup hits them; SSR is a necessity, not a nicety.
VLM reflection does not emerge naturally (unlike text LLMs); Forced Rethinking shows external guidance can be internalized.
Pure RL works: 38K queries, no distillation, no SFT on long CoT—yet beats o1 on MathVista and MathVerse.
Implementation cost is tiny: SSR is a few lines of replay code; Forced Rethinking adds one sentence to rollouts. Training tricks spread far faster than architecture innovations.

7. The Most Surprising Finding: The Model Questions the Questions

In the paper's Figure 3, VL-Rethinker discovers a flaw in the problem itself during reflection—recognizing contradictory premises instead of blindly solving an unsolvable problem. This emergent metacognitive ability was never explicitly taught. Since all current benchmarks assume problems are correct but real-world problems are often flawed, an AI that can question the problem itself is closer to genuine intelligence than one that only solves.

Paper Details

| Item | Content | |:---|:---| | Title | VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning | | Authors | Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen | | Institutions | HKUST, University of Waterloo, INF.AI, Vector Institute | | arXiv ID | 2504.08837 | | Date | 2025-04-10 | | Core contributions | Selective Sample Replay (SSR) for vanishing advantages; Forced Rethinking for VLM self-reflection | | Training data | 38,870 queries (16K/20K after cleaning) | | Best results | MathVista 80.4% (72B), MathVerse 63.5% (72B)—surpassing OpenAI o1 |

VL-Rethinker: Why VLMs Don't Learn to Think on Their Own—and How Forced Rethinking Fixes It

Summary

VL-Rethinker: Why VLMs Don't Learn to Think on Their Own—and How Forced Rethinking Fixes It

1. A Puzzling Observation

2. Vanishing Advantages: When the Model Is Too Strong, the Gradient Dies

3. SSR: Recovering Gold from the Trash Heap of History

4. Forced Rethinking: If the Model Won't Reflect, Make It

5. Results: The VLM Finally Slow-Thinks

6. The Author's Bets

7. The Most Surprising Finding: The Model Questions the Questions

Paper Details

Tags