[论文] Same Beliefs, Different Policies: Learning Reusable Representations fo...

论文概要

研究领域: ML
作者: Jingwei Ni, Dmitry Abulkhanov, Vladislav Smirnov, Markus Leippold, Elliott Ash, Timothy Baldwin et al.
发布时间: 2026-10-24
arXiv: 2610.30450

中文摘要

推理 LLM 常在已经得到正确答案之后继续推理，浪费算力，有时甚至把正确解换成错误解。何时停止是一个元推理问题：模型须依据自己已能回答的内容，权衡继续推理的收益与成本。直接效用训练是在选定的 token 价位上学习停止表示，支持新价位可能要再训练控制器。本文提出 BELT（基于信念的终止）：把结果预测与停止控制分离——先从冻结 LLM 的隐状态学习对当前与未来答案的信念，再在冻结特征上拟合一个线性停止策略。预测目标与 token 价格无关，所以一个探针即可通过 CPU 上的策略重拟合或阈值调整支持不同的准确率–成本权衡。五个分布外数学基准平均：BELT 在 Qwen3-1.7B 上减少 41% 生成 token、在 OLMo-3-7B-Think 上减少 51%，且不损失准确率。同一个数学训练的 BELT 零训练迁移到代码与智能体任务。三个域中其准确率–成本权衡与端到端控制器相当或更优；在 OLMo3 上其探针参数比最强基线 OS-Pruner 的单个检查点少 21.2 倍，训练 GPU 时少 55 倍（后者需五个价格专用检查点）。

原文摘要

Reasoning LLMs often continue reasoning after reaching a correct answer, wasting computation and sometimes replacing it with an incorrect one. Deciding when to stop is a metareasoning problem: the model must weigh what further reasoning could achieve against its cost, given what it can already answer. Direct utility training learns a stopping representation at a chosen token price; supporting new prices can require additional controller training. We introduce BELT (BELief-based Termination), which separates outcome prediction from stopping control. It learns beliefs about current and future answers from a frozen LLM's hidden states, then fits a linear stopping policy on the frozen features. The prediction targets are independent of token price, so one probe supports different accuracy-cost trade-offs through CPU policy refitting or threshold adjustment. Averaged over five out-of-distribution (OOD) math benchmarks, BELT reduces generated tokens by 41% on Qwen3-1.7B and 51% on OLMo-3-7B-Think without loss of accuracy. The same math-trained BELT transfers to coding and agentic tasks without training on code or agentic data. In all three domains, its accuracy-cost trade-offs are competitive with or better than those of end-to-end controllers. On OLMo3, its probe has 21.2x fewer parameters than one checkpoint of the strongest baseline, OS-Pruner, and needs 55x fewer training GPU-hours than OS-Pruner's five price-specific checkpoints.

自动采集于 2026-10-24

#论文 #arXiv #ML #小凯