Paper Overview
Field: Machine Learning Authors: Yingxuan Zhuang, Miao Pan, Wangjie Gan Published: 2026-09-25 arXiv: 2609.21967
Summary
Reinforcement learning is widely used to sharpen reasoning in multimodal large language models (MLLMs), yet its effect on hallucination is uneven. The authors trace this to two weak points in the correction chain from reward to parameter update.
- Rollout level: High-difficulty queries — those with high semantic entropy — frequently produce unanimously wrong sample groups, collapsing the group-relative advantage to zero exactly where hallucination risk is highest.
- Optimization level: Confident-but-wrong tokens are gradient-invisible. The expected score-gradient norm of a categorical policy vanishes as the distribution sharpens, so the predictions that most need correction receive the weakest updates.
- Each branch individually outperforms GRPO.
- The interaction between the two branches is statistically significant on VideoMMMU (+4.0, 95% CI [1.1, 6.9]).
- DEEPO reduces hallucination while preserving accuracy and training stability.
The DEEPO Method
DEEPO (Dual-Entropy Enhanced Policy Optimization) is a two-stage enhancement combining signal variance regularization with gradient preconditioning:
1. Semantic-entropy-triggered expert prefixes: On high-uncertainty queries, grounded continuations are injected to provide direct supervision and restore advantage variance. 2. Advantage-sign-aware Renyi preconditioning: This offsets logit-level saturation, allowing corrections to reach confidently wrong predictions within the operating confidence interval.
Results
*Automatically collected on 2026-09-26.*