Paper Overview
Field: NLP / Multimodal Large Language Models Authors: Wenbo Hu, Xin Chen, Yan Gao-Tian Published: 2025-04-10 arXiv: 2504.07849
Abstract (translated from Chinese summary)
Group Relative Policy Optimization (GRPO) has emerged as the de facto Reinforcement Learning (RL) objective driving recent advancements in Multimodal Large Language Models. However, extending this success to open-source multimodal generalist models remains heavily constrained by two primary challenges: the extreme variance in reward topologies across diverse visual tasks, and the inherent difficulty of balancing fine-grained perception with multi-step reasoning capabilities.
To address these issues, the authors introduce Gaussian GRPO (G2RPO), a novel RL training objective that replaces standard linear scaling with non-linear distributional matching. By mathematically forcing the advantage distribution of any given task to strictly converge to a standard normal distribution N(0,1), G2RPO theoretically ensures inter-task gradient fairness, mitigates vulnerability to heavy-tailed outliers, and provides symmetric updates for positive and negative rewards.
Leveraging the enhanced training stability offered by G2RPO, two task-level shaping mechanisms are introduced to seamlessly balance perception and reasoning:
1. Response-length shaping: dynamically elicits extended reasoning chains for complex queries while enforcing direct outputs to strengthen visual grounding. 2. Entropy shaping: tightly constrains the model's exploration region, effectively preventing both entropy collapse and entropy explosion.
Combining these methods, the authors present OpenVLThinkerV2, a highly robust generalist multimodal model. Extensive evaluation on 18 diverse benchmarks demonstrates that its performance surpasses strong open-source models as well as leading proprietary frontier models.
---
*Automatically collected on 2026-04-12. Source: arXiv:2504.07849*