[论文] Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

论文概要

研究领域: NLP 作者: Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu, Jianguo Zhang, Zhiwei Liu, Tulika Manoj Awalgaonkar, Liangwei Yang, Shelby Heinecke, Silvio Savarese, Huan Wang 发布时间: 2026-09-21 arXiv: 2609.24985

中文摘要

多轮工具使用中的失败可能取决于单次模型调用，但仅凭奖励的变化无法揭示哪一次调用值得训练。当奖励依赖于后续交互时，其变化可能反映的是下游随机性而非当前动作之间的差异。我们提出 Critical-State RL，用于识别多轮交互中可训练的状态。给定任务定义的候选调用和局部奖励，该方法评估每个奖励是否捕捉了动作对任务成功的影响，以及是否存在超越参考策略的改进空间。随后采用嵌套采样将动作依赖的奖励变化与延续噪声分离，并在选定状态上使用上下文老虎机训练优化策略。在 Berkeley Function Calling Leaderboard (BFCL) v4 上的实验比较了在诊断选定状态与替代状态进行训练的效果：对于缺失函数任务，诊断方法选择了工具可用后的响应；对于缺失参数任务，则选择了缺失参数提供前的响应。在选定响应上训练使性能得到提升（缺失函数任务提升约 14 个百分点），而在替代状态上训练则使性能持平或变差。我们还在不同模型和任务上应用了该方案，包括记录重复调用避免和记忆管理等场景。

原文摘要

Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Ca...

#论文 #arXiv #NLP #小凯