ICRL: Learning to Internalize Self-Critique with Reinforcement Learning (arXiv 2505.10885)

Paper Overview

Research area: Machine Learning Authors: Jianbo Lin, Xiaomin Yu, Yi Xin Released: 2025-05-15 arXiv: 2505.10885

Summary

Large language model-based agents make mistakes, yet critique can often guide the same model toward correct behavior. However, when critique is removed, the model may fail again on the same query, indicating that it has not internalized the critique's guidance into its underlying capability. Meanwhile, a frozen critic cannot improve its feedback quality over time, limiting the potential for iterative self-improvement.

To address this, the authors propose ICRL (learning to Internalize Self-Critique with Reinforcement Learning), a novel framework that jointly trains a solver and a critic from a shared backbone to convert critique-induced success into unassisted solver ability. Key components:

• Critic reward from solver gains: The critic is rewarded based on the solver's subsequent performance gain, incentivizing actionable feedback.
• Distribution calibration reweighting: To address the distribution shift between critique-conditioned and critique-free behavior, ICRL selectively transfers critique-guided improvements that are compatible with the solver's own prompt distribution.
• Role-level grouped advantage estimation: This stabilizes the joint optimization across the two roles.

Together, these mechanisms ensure the solver learns to self-improve without external critique rather than becoming dependent on critique-conditioned behavior.

Results

Evaluated on diverse benchmarks spanning agent and mathematical reasoning tasks using Qwen3-4B and Qwen3-8B backbones:

• Consistent improvements over GRPO: +6.4 points on average on agent tasks and +7.0 points on mathematical reasoning.
• Notably, the learned 8B critic performs on par with a 32B critic while using significantly fewer tokens.

--- *Auto-collected on 2026-05-19*