Boosting LLM Exploration via Weak-Model Guidance in RLVR

Forum topic · 小凯 · 2026-08-30

Summary

This paper (arXiv:2608.27420) proposes a simple yet effective method to preserve generative diversity during Reinforcement Learning with Verifiable Rewards (RLVR). RLVR improves LLM reasoning but typically causes policy entropy collapse, narrowing reasoning coverage and degrading pass@k at large k. Existing remedies rely on algorithmic regularization, but the authors introduce a cross-model, non-parametric perturbation: the target model is forced to complete answers from partial reasoning trajectories generated by a smaller, weaker language model. These unfamiliar prefixes disrupt over-confidence and encourage exploration of distinct reasoning paths. Experiments on multiple math benchmarks show the method consistently outperforms vanilla RLVR, with gains growing as k increases—indicating substantially expanded reasoning coverage. The approach mitigates entropy collapse efficiently without extra supervised fine-tuning, complex reward design, or elaborate prompting.

Paper Overview

Research area: NLP Authors: Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao arXiv: 2608.27420

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@ $$k$$ for large $$k$$ . While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language model. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the proposed method on multiple math benchmarks, showing that it consistently outperforms vanilla RLVR. Notably, the performance gains become increasingly evident as $$k$$ grows, demonstrating substantially expanded reasoning coverage. Moreover, our method efficiently mitigates entropy collapse without requiring additional SFT, complex reward design, or elaborate prompting.

Key Points

Problem: RLVR improves reasoning but causes entropy collapse, shrinking exploration coverage and hurting pass@ $$k$$ at large $$k$$ .
Method: Cross-model guidance—force the target model to answer from partial reasoning prefixes produced by a smaller, weaker model, disrupting over-confidence and encouraging diverse reasoning paths.
Results: Consistent improvements over vanilla RLVR on multiple math benchmarks, with gains growing as $$k$$ increases.
Simplicity: No extra SFT, complex reward design, or elaborate prompting needed.

---

*Auto-collected on 2026-08-30*