CORA: Bridging the Thinking-Answer Gap in Multimodal RLVR

Forum topic · 小凯 · 2026-06-16

Summary

Reinforcement learning with verifiable rewards (RLVR) has proven effective at eliciting reasoning in large language models and has been extended to multimodal settings. However, existing approaches mostly emphasize visual coverage of reasoning traces and reducing visual hallucinations, while overlooking semantic inconsistency between the reasoning process and the final answer. This paper investigates thinking-answer inconsistency in RLVR for large vision-language models (LVLMs). Through detailed analyses of rollouts collected during Group Relative Policy Optimization (GRPO) training and post-RLVR evaluation outputs, the authors show the issue persists during training and remains at inference. They propose CORA (Consistency-Oriented Reasoning Alignment), a lightweight plug-and-play consistency reward model that injects thinking-answer semantic consistency into RLVR, paired with Hybrid Reward Advantage Segmentation (HRAS) to stably balance task and consistency optimization. Experiments on representative multimodal reasoning benchmarks across mainstream LVLMs show CORA improves task performance while mitigating inconsistency, yielding more faithful reasoning traces. Paper: arXiv 2606.14691.

Paper Overview

Field: Computational Linguistics (CL)
Authors: Jiayue Cao, Zhicong Lu, Xuehan Sun
Published: 2026-06-12
arXiv: 2606.14691

Abstract

Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models, motivating its extension to multimodal scenarios. Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer.

In this paper, the authors delve into thinking-answer inconsistency in RLVR for large vision-language models (LVLMs). Thorough analyses of rollouts collected throughout the Group Relative Policy Optimization (GRPO) training process and post-RLVR evaluation outputs show that this issue persists during training and remains present during inference.

Method: CORA

Motivated by the analysis, the authors propose Consistency-Oriented Reasoning Alignment (CORA):

A lightweight, plug-and-play consistency reward model introduces thinking-answer semantic consistency into RLVR.
Hybrid Reward Advantage Segmentation (HRAS) further stabilizes the coordination between task optimization and consistency optimization.

Results

Extensive experiments on representative multimodal reasoning benchmarks and mainstream LVLMs demonstrate that CORA improves task performance while effectively mitigating thinking-answer inconsistency, producing more faithful reasoning traces.

---

*Auto-collected on 2026-06-16.*