VL-Rethinker: Forcing Vision-Language Models to Reflect via Pure Reinforcement Learning

Forum topic · 小凯 · 2026-05-11

Summary

A joint team from HKUST, University of Waterloo, and INF.AI introduced VL-Rethinker, a reinforcement-learning-only approach (no distillation) to enable slow-thinking capabilities in vision-language models (VLMs). The work identifies two key problems: unlike text LLMs such as DeepSeek-R1, VLMs trained with standard GRPO do not spontaneously develop self-reflection, and they suffer from severe vanishing advantages, where group-relative gradient signals collapse as training progresses. Two techniques address this: Selective Sample Replay (SSR), a prioritized replay buffer of non-zero-advantage samples that compensates for gradient sparsity, and Forced Rethinking, a training-time intervention that appends rethinking triggers to partial rollouts so that reflection becomes an internalized, spontaneous behavior. Built on Qwen2.5-VL, VL-Rethinker-72B reaches 80.4% on MathVista and 63.5% on MathVerse, surpassing OpenAI o1 without distillation, while VL-Rethinker-7B outperforms open-source 7B multimodal reasoning models such as OpenVLThinker and MM-Eureka. Remaining gaps on MathVision and MMMU-Pro highlight limits in visual perception and data coverage. The paper (arXiv:2504.08837) offers both a diagnosis of why text-LLM RL recipes fail in multimodal settings and practical remedies.

VL-Rethinker: Forcing Vision-Language Models to Reflect — A Reinforcement Learning Path to Multimodal Slow Thinking

> In April 2025, a joint team from HKUST, University of Waterloo, and INF.AI proposed a method to enhance slow-thinking capabilities in vision-language models (VLMs) through pure reinforcement learning (no distillation). The work reveals a key phenomenon: unlike text LLMs, VLMs do not spontaneously develop self-reflection under standard GRPO training and suffer from severe vanishing advantages. By introducing Selective Sample Replay (SSR) and Forced Rethinking, VL-Rethinker surpasses OpenAI o1 on MathVista (80.4%) and MathVerse (63.5%), achieving open-source state of the art on multiple multimodal benchmarks.

---

1. Background: The Multimodal Slow-Thinking Gap

Since 2024, "slow-thinking" systems like OpenAI o1 and DeepSeek-R1 have made breakthroughs in text-based math and science reasoning. However, their multimodal reasoning remains comparable to "fast-thinking" models (GPT-4o, Qwen2.5-VL):

| Model | MathVista | MathVerse | MathVision | |:---:|:---:|:---:|:---:| | GPT-4o | 60.0% | 41.2% | 30.6% | | OpenAI o1 | 73.9% | 57.0% | 60.3% | | Qwen2.5-VL-72B | 74.8% | 57.2% | 38.1% |

> Key observation: o1 improves over GPT-4o by 30%+ on text-only math benchmarks, but on multimodal benchmarks it merely matches — or falls slightly below — Qwen2.5-VL. Multimodal slow thinking has not emerged naturally alongside text slow thinking.

Wang et al. (2025) formalize the question: how can VLM slow thinking be trained directly with RL, without relying on distillation?

---

2. Structural Flaws of GRPO in Multimodal Settings

2.1 Quantifying Vanishing Advantages

Group Relative Policy Optimization (GRPO) estimates advantages via group-normalized rewards:

\hat{A}_{i,t} = \frac{r(x, y_i) - \text{mean}(\{r(x, y_j)\}_{j=1}^{G})}{\text{std}(\{r(x, y_j)\}_{j=1}^{G})}

Wang et al. found that during Qwen2.5-VL-72B training, the effective gradient signal decays persistently:

| Training progress | Effective queries (non-zero advantage) | Trend | |:---:|:---:|:---:| | Initial | ~40% | — | | After 16×16 gradient steps | < 20% | ↓ 50% |

> Causal analysis: as the policy improves, within-group samples become homogeneous (all correct or all wrong), driving both the numerator and standard deviation toward zero. This is especially severe for a 72B-scale VLM whose accuracy saturates quickly on training data.

2.2 Key Differences vs. Text LLMs

| Dimension | Text LLM (DeepSeek-R1) | VLM (Qwen2.5-VL) | |:---|:---|:---| | Spontaneous slow thinking | ✅ common | ❌ rare | | Vanishing advantages severity | moderate | severe | | Training data volume | millions | typically small (38K–200K) | | Reward signal | relatively clean | vision-language interaction is noisier |

> Hypothesis: multimodal reward signals may be blurred by visual-understanding noise, making it hard for the model to build a clear causal link between "reflection → better outcome."

---

3. Selective Sample Replay: Dynamic Curriculum and Experience Replay

3.1 Mechanism

SSR maintains a replay buffer $\mathcal{B}_{\text{replay}}$ that stores only non-zero-advantage samples, sampled with priority proportional to the absolute advantage:

P(\text{select } j) = \frac{|\hat{A}_j|^{\alpha}}{\sum_{k \in \mathcal{B}_{\text{replay}}} |\hat{A}_k|^{\alpha}}

where $\alpha$ controls priority strength. Design rationale:

1. Keep decision-boundary samples: large-correct or large-wrong samples lie at the model's capability boundary and are most informative 2. Dynamic curriculum: online prioritized replay of hard samples automatically shifts training focus 3. Compensate gradient sparsity: when the current batch lacks effective samples, replenish from the buffer

3.2 Relation to Prior Work

| Method | Mechanism | Setting | Relation to SSR | |:---|:---|:---|:---| | Prioritized Experience Replay (PER) | TD-error priority | off-policy RL | inspiration; SSR applies it to on-policy GRPO | | Curriculum Learning | static/offline difficulty ordering | supervised learning | SSR is an online dynamic version | | Rejection Sampling | filters low-quality samples | SFT/RL | SSR retains and prioritizes valuable hard samples |

3.3 Ablation

| Config | MathVision | MathVista | MathVerse | MMMU-Pro | |:---:|:---:|:---:|:---:|:---:| | Vanilla GRPO | 26.0% | 70.9% | 51.4% | 38.8% | | + Filter (drop zero-advantage) | 28.5% | 72.0% | 50.0% | 40.0% | | + SSR (prioritized replay) | 32.3% | 74.9% | 54.2% | 41.7% |

> SSR outperforms simple filtering by an additional 3.8 points on MathVision — prioritized sampling beats mere retention.

---

4. Forced Rethinking: From External Scaffolding to Internalized Reflection

4.1 Motivation

Even with stable training, VLMs do not spontaneously reflect — unlike text LLMs, which often exhibit self-reflection patterns ("Wait", "Alternatively") under RL. Forced Rethinking's core hypothesis: reflection can be induced by external intervention during training, then internalized as spontaneous behavior.

4.2 Implementation

For a fraction of rollouts ( $$q < 1$$ ):

1. The model generates an initial answer $$y_1$$ 2. A rethinking trigger is appended: $y = y_1 \oplus \text{trigger} \oplus y_2$ 3. Only trajectories whose final answers are correct are kept 4. An additional SFT loss is applied to successful trajectories

Triggers fall into three types:

Self-verification: check correctness of reasoning steps
Self-correction: identify and fix errors
Self-questioning: question premises and assumptions

> Key distinction: unlike S1 (Muennighoff et al., 2025), which uses inference-time budget forcing, Forced Rethinking is a training intervention. At inference time no trigger is needed — the model has learned to reflect spontaneously when necessary.

4.3 Emergent Metacognition

The paper reports a striking case: during reflection, VL-Rethinker identified a flaw in the problem itself (e.g., contradictory conditions) rather than blindly solving it. This suggests something beyond simple answer correction — metacognition: monitoring and evaluating its own cognitive process and the validity of the question itself.

4.4 Ablation

| Config | MathVision | MathVista | EMMA | |:---:|:---:|:---:|:---:| | SSR only | 29.8% | 72.4% | 29.5% | | SSR + Forced Rethinking | 32.3% | 74.9% | 29.7% |

Forced Rethinking adds 2.5 points on MathVista and, more importantly, grants the model a transferable reflection ability.

---

5. Results: Leading from 7B to 72B

5.1 72B Scale: Partially Surpassing Closed-Source SOTA

| Benchmark | OpenAI o1 | Qwen2.5-VL-72B | VL-Rethinker-72B | |:---:|:---:|:---:|:---:| | MathVista | 73.9% | 74.8% | 80.4% | | MathVerse | 57.0% | 57.2% | 63.5% | | MathVision | 60.3% | 38.1% | 44.9% | | MMMU-Pro | 62.4% | 51.6% | 55.9% | | EMMA | 45.7% | 34.1% | 38.5% | | MEGA-Core | 56.2% | 49.0% | 51.3% |

> On math reasoning benchmarks (MathVista, MathVerse), VL-Rethinker-72B surpasses OpenAI o1 — achieved via pure RL, with no distilled o1 data.

5.2 7B Scale: Best Among Open Models

| Benchmark | OpenVLThinker-7B | MM-Eureka-7B | VL-Rethinker-7B | |:---:|:---:|:---:|:---:| | MathVista | 70.2% | 73.0% | 74.9% | | MathVerse | 47.9% | 50.3% | 54.2% | | MathVision | 25.3% | 26.9% | 32.3% | | MMMU-Pro | 37.3% | — | 41.7% |

> VL-Rethinker-7B leads all open 7B multimodal reasoning models. OpenVLThinker uses distillation and MM-Eureka is based on InternLM — VL-Rethinker surpasses them with pure RL and two training techniques.

---

6. Deeper Analysis: Data, Stability, and Generalization

6.1 Data Composition

Ablations (Table 3) show:

Removing text-only queries: minor impact
Removing science queries: significant performance drop
Conclusion: domain diversity in science is critical for general reasoning

6.2 Training Dynamics

The team started from 38,870 queries; after cleaning, per-scale counts:

7B model: ~16,000 queries
32B/72B models: ~20,000 queries

Training saturates within 3 epochs; SSR's replay buffer is cleared every episode (1024 queries).

6.3 vs. Distillation Approaches

| Method | Data source | Training | MathVista (7B) | |:---|:---|:---|:---:| | OpenVLThinker | distilled from o1/R1 | SFT | 70.2% | | R1-OneVision | distillation + RL | mixed | 64.1% | | VL-Rethinker | own 38K queries | pure RL | 74.9% |

> The pure-RL route avoids dependence on a teacher model and outperforms distillation — a viable path for autonomous scaling of VLM reasoning.

---

7. Limitations and Open Questions

7.1 Gap on MathVision

Despite surpassing o1 on MathVista and MathVerse, VL-Rethinker trails on MathVision (44.9% vs 60.3%) and MMMU-Pro (55.9% vs 62.4%), likely reflecting insufficient training-data coverage on those benchmarks and the limits of visual perception.

7.2 Generality of Forced Rethinking

Forced Rethinking was validated only on math and science. More open-ended multimodal tasks (visual dialogue, creative generation) may require redesigned triggers and reflection formats.

7.3 Root-Cause Fixes for Vanishing Advantages

SSR treats the symptom rather than the cause (the binary reward mechanism). Future directions include:

Fine-grained process rewards
Continuous rewards replacing binary rewards
Dynamic group-size adjustment

---

8. Conclusion

VL-Rethinker provides a distillation-free, pure-RL path to multimodal slow thinking: SSR resolves training instability, Forced Rethinking induces reflection, and the resulting models match or exceed the state of the art on multiple benchmarks.

The deeper lesson: text LLMs and VLMs differ fundamentally in RL dynamics. Directly transplanting text-model training recipes to multimodal settings can cause severe optimization failures (e.g., vanishing advantages). VL-Rethinker offers both a diagnosis and practical solutions for this problem.

---

Paper Details

| Item | Content | |:---|:---| | Title | VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning | | Authors | Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen | | Institutions | HKUST, University of Waterloo, INF.AI, Vector Institute | | arXiv ID | 2504.08837 | | Date | 2025-04-10 | | Core contributions | Selective Sample Replay (SSR); Forced Rethinking; pure-RL multimodal slow thinking | | Training data | 38,870 queries (16K–20K after cleaning) | | Key results | MathVista 80.4% (72B), MathVerse 63.5% (72B) — surpassing OpenAI o1 | | Project page | https://tiger-ai-lab.github.io/VL-Rethinker/ |