Boosting LLM Exploration via Weak-Model Guidance in RLVR

Forum topic · 小凯 · 2026-08-30

Summary

Researchers propose a simple method to preserve generative diversity in large language models during Reinforcement Learning with Verifiable Rewards (RLVR). RLVR typically improves reasoning but causes policy entropy collapse, narrowing reasoning coverage and degrading pass@k at large k. Instead of relying on internal exploration or algorithmic regularization, the approach forces the target model to complete answers from partial reasoning trajectories generated by a smaller, weaker language model. These unfamiliar prefixes disrupt over-confidence and push the model toward diverse reasoning paths. Experiments across multiple math benchmarks show consistent improvements over vanilla RLVR, with gains growing as k increases, indicating substantially broader reasoning coverage. The method mitigates entropy collapse without extra supervised fine-tuning, complex reward design, or intricate prompting.

Paper Overview

Field: NLP
Authors: Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
arXiv: 2608.27420

Background

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning ability, but it often causes a drop in policy entropy. This entropy collapse narrows reasoning coverage and degrades pass@ $$k$$ performance at large $$k$$ . Existing methods mitigate this via algorithmic regularizations, but cross-model non-parametric perturbation has been neglected.

Method

The authors propose a simple yet effective approach to preserve generative diversity during RLVR:

Instead of relying solely on the target model's internal exploration, the target model is forced to generate answers based on partial reasoning trajectories produced by a smaller, weaker language model.
These unfamiliar prefixes effectively disrupt over-confidence and encourage exploration of distinct reasoning paths.

Results

On multiple mathematical benchmarks, the method consistently outperforms vanilla RLVR.
Performance gains become more pronounced as $$k$$ grows, demonstrating substantially expanded reasoning coverage.
The method efficiently mitigates entropy collapse without requiring additional SFT, complex reward design, or elaborate prompting.

Summary (translated from the original Chinese post)

RLVR significantly boosts LLM reasoning but typically reduces policy entropy, shrinking reasoning coverage and hurting pass@ $$k$$ at large $$k$$ . Rather than internal exploration or algorithmic regularization, this work introduces cross-model perturbation: the target model completes answers from partial reasoning trajectories sampled by a smaller, weaker model. These unfamiliar prefixes break over-confidence and encourage diverse reasoning. Experiments show consistent gains over vanilla RLVR on math benchmarks, with benefits amplifying at larger $$k$$ .

---

*Auto-collected on 2026-08-30*