Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

Forum topic · 小凯 · 2026-09-23

Summary

Critical-State RL is a new reinforcement learning method that identifies which specific model calls in a multi-turn tool-use interaction are actually trainable. When rewards depend on later interactions, their variation can reflect downstream randomness rather than the quality of the current action, so reward deltas alone cannot reveal which step deserves training. Given task-defined candidate calls and local rewards, the method evaluates whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible, then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states with contextual-bandit training. Experiments on Berkeley Function Calling Leaderboard (BFCL) v4 show that training at the diagnosed states yields large gains—for missing-function tasks about 14 percentage points—whereas training at alternative states produces flat or worse performance. The approach also generalizes across models and tasks, including scenarios such as avoiding repeated calls and memory management. Paper: arXiv:2609.24985.

Overview

Field: NLP
Authors: Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu, Jianguo Zhang, Zhiwei Liu, Tulika Manoj Awalgaonkar, Liangwei Yang, Shelby Heinecke, Silvio Savarese, Huan Wang
Published: 2026-09-21
arXiv: 2609.24985

Key Points

Multi-turn tool-use failures can hinge on a single model call, but reward variation alone does not reveal which call would benefit from training.
When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions.
Critical-State RL identifies trainable states in multi-turn interactions: given task-defined candidate calls and local rewards, it assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible.
Nested sampling separates action-dependent reward variation from continuation noise.
The policy is optimized at the selected states using contextual-bandit training.

Experiments

Evaluated on the Berkeley Function Calling Leaderboard (BFCL) v4, comparing training at diagnosed states versus alternative states.
For the missing-function task, diagnosis selected the response after tools became available; for the missing-parameter task, it selected the response before missing parameters were provided.
Training at the selected responses improved performance (about 14 percentage points on the missing-function task), while training at alternative states led to flat or worse results.
The scheme was also applied across different models and tasks, including avoiding repeated calls and memory management.

Abstract (original)

Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Calling Leaderboard (BFCL) v4...

---

*Auto-collected on 2026-09-23*