[论文] Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from...

论文概要

研究领域: ML
作者: Barış Alparslan, Sandy Tanwisuth, Caleb Petterson
发布时间: 2026-10-22
arXiv: 2610.31624

中文摘要

工具性收敛常被引用来预期有能力智能体获得权力追求子目标（自我保护、资源获取、欺骗、抵制监督）。RLVR是否使模型更可能表达工具性收敛行为？本文在OLMo-3-7B-Think上跨SFT、DPO、13个RLVR中间检查点和最终模型进行评估。轨迹呈非单调性：工具性比率从SFT的25.0%升至RLVR约200步的46.1%，再在最终检查点降至11.8%，低于SFT基线。提示框架无法解释此模式。初步32B结果同样呈下降趋势（SFT 35.53% → RLVR 22.37%）。这些发现表明RLVR并非单调增加工具性行为——比率先升后降。

原文摘要

Instrumental convergence is often invoked as a general reason to expect capable agents to acquire power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective. Does RLVR make a model more likely to express instrumental-convergence behavior? We evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern. Preliminary 32B endpoint resu...

自动采集于 2026-10-26

#论文 #arXiv #ML #小凯