[论文] CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR vi...

论文概要

研究领域: CL
作者: Jiayue Cao, Zhicong Lu, Xuehan Sun
发布时间: 2026-06-12
arXiv: 2606.14691

中文摘要

可验证奖励强化学习（RLVR）成功激发了大语言模型的推理能力，促使其扩展到多模态场景。现有方法主要关注提高推理轨迹的视觉覆盖和缓解视觉幻觉，但低估了推理过程与最终答案之间的语义不一致性。在本文中，我们深入研究了大型视觉语言模型（LVLMs）中RLVR的思考-答案不一致问题，通过对GRPO训练过程中收集的rollouts和RLVR后评估输出的 thorough 分析表明，该问题在训练期间持续存在并在推理期间仍然出现。受分析启发，我们提出了面向一致性的推理对齐（CORA），通过轻量即插即用的一致性奖励模型将思考-答案语义一致性引入RLVR，并进一步结合混合奖励优势分割（HRAS）以稳定协调任务和一致性优化。在代表性多模态推理基准和主流LVLMs上的广泛实验表明，CORA提高了任务性能，同时有效缓解了思考-答案不一致，产生了更忠实的推理轨迹。

原文摘要

Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models, motivating its extension to multimodal scenarios. Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer. In this paper, we delve into thinking-answer inconsistency in RLVR for large vision-language models (LVLMs), showing thorough analyses of rollouts collected throughout Group Relative Policy Optimization (GRPO) training process and post-RLVR evaluation outputs that this issue persists during training and remains present during inference. Motivated by the analysis, we propose Consistency-Oriente...

自动采集于 2026-06-16

#论文 #arXiv #CL #小凯