VL-Rethinker: Forcing Vision-Language Models to Reflect via Pure Reinforcement Learning
In April 2025, a joint team from HKUST, the University of Waterloo, and INF.AI proposed a method to enhance the slow-thinking ability of vision-language models (VLMs) through pure reinforcement learning, without any distillation. The work reveals a key phenomenon: unlike text LLMs, VLMs do not spontaneously exhibit self-reflection under standard GRPO training, and they face a severe *vanishing advantages* problem. By introducing Selective Sample Replay (SSR) and Forced Rethinking, VL-Rethinker surpasses OpenAI o1 on MathVista (80.4%) and MathVerse (63.5%) and achieves state-of-the-art open-source results across multiple multimodal benchmarks.
Key points
- Multimodal slow thinking did not emerge naturally. While OpenAI o1 gains 30%+ over GPT-4o on text math benchmarks, on multimodal benchmarks it only matches Qwen2.5-VL-72B (e.g., 73.9% vs 74.8% on MathVista), showing slow thinking does not transfer automatically to multimodal settings.
- Vanishing advantages under GRPO. Advantage estimates under GRPO are group-normalized rewards; as the policy improves, group samples become all-correct or all-wrong, so both numerator and standard deviation approach zero. During Qwen2.5-VL-72B training, effective (non-zero advantage) queries dropped from ~40% to under 20% within 16×16 gradient steps.
- Selective Sample Replay (SSR). A replay buffer stores only non-zero-advantage samples and samples them with priority proportional to |advantage|^α, retaining decision-boundary samples, providing dynamic online curriculum learning, and compensating for gradient sparsity. Ablations show SSR beats simple zero-advantage filtering by 3.8 points on MathVision (32.3% vs 28.5%).
- Forced Rethinking. For a fraction q of rollouts, the model generates an initial answer, an external rethinking trigger (self-verification, self-correction, or self-questioning) is appended, and only trajectories whose final answers are correct are kept, with an additional SFT loss. Unlike S1's inference-time budget forcing, this is a training intervention: the model internalizes reflection and reflects spontaneously at inference time.
- Emergent metacognition. The authors report a case where VL-Rethinker, during reflection, identifies a flaw in the problem itself (contradictory conditions) rather than blindly solving it—evaluating problem validity rather than merely correcting answers.
- Training uses 38,870 initial queries (cleaned to ~16K for 7B, ~20K for 32B/72B), saturating within 3 epochs; the SSR replay buffer clears every episode of 1024 queries.
- Ablations show science queries matter: removing them causes significant performance drops, while removing text-only queries has little effect.
- Compared with distillation pipelines (OpenVLThinker: 70.2% MathVista at 7B; R1-OneVision: 64.1%), pure RL with a self-built 38K-query set reaches 74.9%.
- Gaps remain versus o1 on MathVision (44.9% vs 60.3%) and MMMU-Pro (55.9% vs 62.4%), likely due to training-data coverage and visual understanding limits.
- Forced Rethinking was validated only on math/science tasks; open-ended multimodal tasks may need redesigned triggers.
- SSR mitigates rather than cures vanishing advantages; future directions include process rewards, continuous rewards, and dynamic group sizes.
- Title: VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
- Authors: Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen
- Affiliations: HKUST, University of Waterloo, INF.AI, Vector Institute
- arXiv: 2504.08837 (2025-04-10)
- Project page: https://tiger-ai-lab.github.io/VL-Rethinker/
Main results
At the 72B scale (pure RL, no distillation from o1):
| Benchmark | OpenAI o1 | Qwen2.5-VL-72B | VL-Rethinker-72B | |:---:|:---:|:---:|:---:| | MathVista | 73.9% | 74.8% | 80.4% | | MathVerse | 57.0% | 57.2% | 63.5% | | MathVision | 60.3% | 38.1% | 44.9% | | MMMU-Pro | 62.4% | 51.6% | 55.9% | | EMMA | 45.7% | 34.1% | 38.5% |
At the 7B scale, VL-Rethinker-7B leads open 7B multimodal reasoners (MathVista 74.9%, MathVerse 54.2%, MathVision 32.3%, MMMU-Pro 41.7%), outperforming OpenVLThinker-7B (distillation-based) and MM-Eureka-7B.
Training details and analysis
Limitations
Conclusion
VL-Rethinker provides a distillation-free, pure-RL path to multimodal slow thinking: SSR stabilizes training while Forced Rethinking induces reflection. The deeper lesson is that text LLMs and VLMs differ fundamentally in RL dynamics—naively porting text-side training recipes to multimodal settings can cause serious optimization failures.