Paper Overview
- Area: cs.AI
- Authors: Bin Lei, Yu Li, Prafulla Kumar Choubey, Jiaxin Zhang, Becky Xiangyu Peng, Qinyuan Ye, Kartik Narayan, Caiwen Ding, Silvio Savarese, Chien-Sheng Wu
- Published: 2026-09-13
- arXiv: 2609.11061
- Three instantiations, none needing step-level supervision, span access levels:
- A black-box probe
- A logit-lens depth profile
- A learned activation direction (fit offline, used only in validation before RL training)
- The signal only places forks; the probe costs about \(1\%\) of step compute on mathematics and under \(5\%\) on code when run inside the rollout engine.
- Against Monte-Carlo value curves, the belief-shift signal ranks first in each of the eight model × benchmark panels, ahead of entropy, structural, and LLM-judge baselines.
- Across three model families and two domains, belief-shift forking leads every mathematics aggregate:
- On OLMo-3-7B: +2.6 aggregate and +2.9 on AIME 2026 over the strongest baseline.
- Sweeps every OLMo code column: +6.5 on LiveCodeBench-medium.
Abstract
Tree-structured rollouts give critic-free reinforcement learning with verifiable rewards (RLVR) step-level credit: fork a chain at an intermediate point, and sibling outcome differences estimate step value. Each fork adds sampling cost, so realistic budgets typically allow only a few forks per chain. A fork placed where the outcome is already largely settled yields siblings that mostly agree and provide almost no credit signal; hence, for a given tree size, where forks are placed largely determines how much step-level RL can gain. Most existing mainstream methods place forks by structure, such as fixed lengths, midpoints, and delimiters, or by next-token entropy.The authors formalize fork placement as locating the pivots of the chain's value curve—where the expected outcome turns—and propose belief-shift branching: read the model's answer belief at candidate boundaries and fork just before the step where consecutive beliefs diverge most.