[论文] ICRL: Learning to Internalize Self-Critique with Reinforcement Learnin...

论文概要

研究领域: ML 作者: Jianbo Lin, Xiaomin Yu, Yi Xin 发布时间: 2025-05-15 arXiv: 2505.10885

中文摘要

基于大语言模型的智能体也会犯错，但批评往往可以引导同一模型走向正确行为。然而，当批评被移除时，模型可能在同一查询上再次失败，表明它尚未将批评的指导内化为其底层能力。与此同时，冻结的批评者无法随时间改善其反馈质量，限制了迭代自我改进的潜力。为解决此问题，我们提出通过强化学习内化自我批评（ICRL），一种新颖的框架，从共享主干联合训练求解器和批评者，将批评诱导的成功转化为无辅助的求解器能力。批评者基于求解器随后的性能增益获得奖励，激励可操作的反馈。为解决批评条件和无批评行为之间的分布偏移，ICRL引入分布校准重加权比，选择性地传输与求解器自身提示分布兼容的批评引导改进。此外，角色级分组优势估计稳定了两个角色间的联合优化。这些机制共同确保求解器学会在没有外部批评的情况下自我改进，而不是变得依赖批评条件行为。我们在涵盖智能体和数学推理任务的多样化基准上评估ICRL，使用Qwen3-4B和Qwen3-8B作为主干。结果显示持续改进，在智能体任务上平均比GRPO高6.4分，在数学推理上高7.0分。值得注意的是，学习到的8B批评者与32B批评者相当，同时使用的token显著更少。代码可在该https URL获取。

原文摘要

Large language model-based agents make mistakes, yet critique can often guide the same model toward correct behavior. However, when critique is removed, the model may fail again on the same query, indicating that it has not internalized the critique's guidance into its underlying capability. Meanwhile, a frozen critic cannot improve its feedback quality over time, limiting the potential for iterative self-improvement. To address this, we propose learning to internalize self-critique with reinforcement learning(ICRL), a novel framework that jointly trains a solver and a critic from a shared backbone to convert critique-induced success into unassisted solver ability. The critic is rewarded based on the solver's subsequent performance gain, incentivizing actionable feedback. To address the di...

自动采集于 2026-05-19

#论文 #arXiv #ML #小凯