ICRL: Learning to Internalize Self-Critique with Reinforcement Learning (arXiv 2505.10885)

Paper Overview

Field: Machine Learning Authors: Jianbo Lin, Xiaomin Yu, Yi Xin Published: 2025-05-15 arXiv: 2505.10885

Summary

Large language model-based agents make mistakes, yet critique can often guide the same model toward correct behavior. However, when critique is removed, the model may fail again on the same query, indicating it has not internalized the critique's guidance into its underlying capability. Meanwhile, a frozen critic cannot improve its feedback quality over time, limiting iterative self-improvement.

To address this, the authors propose ICRL (learning to Internalize Self-Critique with Reinforcement Learning), a framework that jointly trains a solver and a critic from a shared backbone to convert critique-induced success into unassisted solver ability.

Key Contributions

• Reward design: The critic is rewarded based on the solver's subsequent performance gain, incentivizing actionable feedback.
• Distribution-calibrated reweighting ratio: Addresses the distribution shift between critique-conditioned and critique-free behavior by selectively transferring critique-guided improvements compatible with the solver's own prompt distribution.
• Role-level grouped advantage estimation: Stabilizes the joint optimization between the two roles.

Together, these mechanisms ensure the solver learns to self-improve without external critique rather than becoming dependent on critique-conditioned behavior.

Results

Evaluated on diverse benchmarks covering agent and mathematical reasoning tasks with Qwen3-4B and Qwen3-8B backbones:

• Average improvement of 6.4 points over GRPO on agent tasks
• Average improvement of 7.0 points on math reasoning
• The learned 8B critic is comparable to a 32B critic while using significantly fewer tokens

Code is available at the link provided in the paper (see the arXiv abstract page).