[论文] Reinforcement Learning with Visually Attributable On-Policy Self-Disti...

论文概要

研究领域: CV 作者: Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang 发布时间: 2026-10-19 arXiv: 2610.26856

中文摘要

基于可验证奖励的多模态强化学习（RLVR）提供了可靠的任务级监督，但在视觉推理中定位错误方面指导有限。在策略自蒸馏（OPSD）通过来自特权教师的密集token级校正来补充这种监督。然而，由此产生的校正可能将视觉证据与由语言先验、回答风格或特权视角本身引起的非视觉变化混为一谈。我们将这一现象形式化为特权校正污染。我们提出视觉归因门控（VAG），使用验证器导出的优势作为优化锚点，并通过证据完整与证据退化教师视角之间的对比来验证每个特权校正。通过仅接受视觉对比在方向上支持的校正，VAG在抑制不一致残差的同时保留了视觉grounding的指导。在六个视觉推理基准测试上的实验表明，VAG一致优于GRPO和OPSD方法，建立了新的最优平均性能。

原文摘要

Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous t...

#论文 #arXiv #CV #小凯