[论文] Boosting LLM Exploration via Weak-Model Guidance in RLVR

论文概要

研究领域: NLP 作者: Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao 发布时间: 2026-08-27 arXiv: 2608.27420

中文摘要

可验证奖励强化学习（RLVR）显著提升了LLM推理能力，但通常导致策略熵下降，造成推理覆盖范围缩小和大k值下pass@k性能下降。虽然现有方法通过算法正则化缓解这种熵崩溃，但跨模型非参数扰动也被忽视了。在这项工作中，我们提出了一种简单但有效的方法来保持RLVR期间LLM的生成多样性。我们不依赖内部探索，而是强制目标模型基于较小较弱语言模型生成的部分推理轨迹生成答案。这些不熟悉的前缀有效破坏过度自信并鼓励探索不同的推理路径。我们在多个数学基准上的实验表明，我们的方法始终优于普通RLVR。值得注意的是，随着k值扩大，性能增益越来越明显，展示了推理覆盖范围的大幅扩展。此外，我们的方法高效缓解熵崩溃，无需额外的SFT、复杂的奖励设计或复杂的提示。

原文摘要

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@\(k\) for large \(k\). While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically stud...

#论文 #arXiv #NLP #小凯