Weak Models as Exploration Coaches: A Counterintuitive Fix for Entropy Collapse in RLVR Training

Forum topic · ✨步子哥 · 2026-08-29

Summary

A Chinese tech forum post discusses the paper 'Boosting LLM Exploration via Weak-Model Guidance in RLVR' (arXiv:2608.27420), which tackles entropy collapse in Reinforcement Learning with Verifiable Rewards (RLVR). Standard GRPO training on Qwen2.5-7B raises pass@1 from 27.83% to 38.29% but reduces pass@128 from 72.15% to 67.76%, as the model converges on a few high-reward reasoning paths. The proposed Prefix-Completion RLVR injects truncated reasoning prefixes from small cross-family models (LLaMA-3.2-1B, Gemma-2-2B) before the target model completes the solution. Counterintuitively, most prefixes are wrong or unhelpful, yet they restore pass@128 to 70.71% while slightly improving pass@1. Analysis shows the value comes from distributional discrepancy rather than correctness: unfamiliar, often mistaken starting points force the model into states it would never explore, delaying entropy collapse. Ablations confirm cross-family prefixes outperform same-family ones, entropy-based truncation helps mildly, and a prefix injection probability of p=0.2 works best. The method requires no reward redesign or algorithm changes, making it a nearly zero-cost fix for the performance-diversity trade-off in RLVR.

A Strange Experiment

Imagine training a math prodigy (Qwen2.5-7B) on competition problems. The standard approach is reinforcement learning — reward correct answers, penalize wrong ones. After several hundred training rounds, her single-attempt accuracy jumped from 27.8% to 38.3%.

But there's a side effect: she increasingly solves problems only one way. If you let her try 128 times, coverage drops from 72.2% before training to 67.8% after. Training made her stronger, but also narrower.

This is the "entropy collapse" problem in RLVR (Reinforcement Learning with Verifiable Rewards): the model early on converges to a few high-reward reasoning paths and never explores other possibilities afterward.

Now researchers have proposed a counterintuitive solution: let a weaker small model (e.g., Gemma-2-2B) do a few reasoning steps first, then feed those steps as an "opening" to the large model and let it continue.

It sounds absurd — having the struggling student coach the top student? But the result: pass@128 improved from 67.8% to 70.7%, while pass@1 barely changed. Even more surprisingly, most of the small model's "openings" are wrong.

The paper is arXiv:2608.27420, "Boosting LLM Exploration via Weak-Model Guidance in RLVR."

Entropy Collapse: Why "Stronger" and "Narrower" Are Twins

To understand the problem, you first need to know how RLVR trains.

Given a math problem $$q$$ , the model generates a reasoning trajectory $$r$$ , a verifier checks the answer, and a reward $$R(q, r)$$ is assigned. The training objective maximizes expected reward. Essentially, it signals to the model: "take the paths that score high more often."

The problem: once the model finds a path that reliably scores, it keeps reinforcing it. Probability mass rapidly concentrates on that path, and other paths get pushed near zero. From an information-theoretic view, this is the policy's entropy plummeting.

The cost of entropy collapse? Loss of diversity. The model becomes a one-trick player — strong at that one trick, but completely stuck on problems it can't solve. pass@1 goes up, but pass@k (the probability of at least one correct answer in k tries) actually drops at large k.

Prior solutions work from inside the algorithm: entropy regularization, KL coefficient tuning, reward shaping. This paper takes a different angle — injecting perturbation from outside the model.

Cross-Model Prefixes: Letting a "Stranger" Start

The core method is called Prefix-Completion RLVR, and the idea is extremely simple:

1. Pick a small model from a different family than the target (e.g., LLaMA-3.2-1B or Gemma-2-2B) 2. Have it generate reasoning for a problem, then randomly truncate at some step to form a "prefix" 3. Append this prefix to the problem and let the large model continue 4. Train the large model with standard GRPO

Formally, standard RLVR generates a full solution from scratch: $r \sim \pi_\theta(\cdot | q)$ . Prefix-Completion instead asks the model to complete the remainder given a prefix $\tilde{r}$ : $r_{\text{suf}} \sim \pi_\theta(\cdot | q, \tilde{r})$ .

A probability $$p$$ controls how often prefixes are injected during training. $$p=0$$ is standard GRPO; $$p=1$$ means every sample has a prefix. The paper finds $$p=0.2$$ is a good balance.

There's also a detail called Entropy-Based Prefix Truncation: rather than randomly picking step L to cut, it looks at the small model's entropy at each step. Cutting where entropy is low (where the model is confident) preserves the part the small model is "sure" of — even if that confidence is wrong.

Why It Works: Perturbation Matters More Than Correctness

This is the paper's most counterintuitive finding.

Researchers used DeepSeek-V4-Flash to evaluate prefix quality, classifying them into four types:

(a) No substantive guidance: the prefix offers no useful reasoning direction
(b) Fully wrong and misleading: the prefix leads the model astray
(c) Flawed but acceptable: partially useful despite errors
(d) Fully correct and appropriate: sound reasoning

The result is startling: most prefixes fall into (a) or (b) — either useless or outright wrong. For LLaMA-3.2-1B prefixes, "fully wrong and misleading" exceeds 50%.

Yet it works. Why?

The answer lies in the nature of exploration. Exploration doesn't need "correct guidance" — it needs "different starting points." Because prefixes come from a different family with far fewer parameters, their reasoning style and distribution differ sharply from the large model's. This distributional discrepancy forces the large model to continue from an unfamiliar state — even if that state is "wrong," it opens a path the large model would never take on its own.

The paper's training dynamics analysis confirms this:

Policy entropy: models trained with prefixes keep notably higher entropy early on, delaying collapse
Average reward: prefix-trained models have lower average reward — the "cost of exploration," since prefixes often mislead
Zero-reward ratio: prefix-trained models have more zero-reward samples — they're attempting harder paths
Non-trivial sample ratio: prefix-trained models have more samples that are neither fully right nor fully wrong — they're exploring boundary regions

It's like sending a chess grandmaster to a tournament where an amateur plays the opening moves of every game. Those openings may be bad, but the grandmaster is forced to find her way out of unfamiliar positions — which trains adaptability. The value of exploration isn't in how good the guide is, but in how new the road is.

The Numbers

Main experiments ran on Qwen2.5-7B and Qwen2.5-Math-7B, with small models LLaMA-3.2-1B and Gemma-2-2B, tested on 6 math reasoning benchmarks (AIME 2024/2025, AMC 2023, MATH 500, Minerva, Olympiad Bench).

Core results (Qwen2.5-7B):

| Method | pass@1 | pass@64 | pass@128 | |--------|--------|---------|----------| | Base model | 27.83 | 67.98 | 72.15 | | + GRPO | 38.29 | 65.02 | 67.76 | | + Gemma-2-2B prefix | 39.01 | 66.78 | 70.71 |

Note pass@128: standard GRPO drops it from 72.15 to 67.76 (entropy collapse), while prefix injection recovers it to 70.71. pass@1 also improves slightly.

Ablations are interesting too:

Same-family prefixes: using Qwen2.5-1.5B or Qwen2.5-7B itself to generate prefixes works clearly worse than cross-family small models. This confirms "distributional discrepancy" is key — prefixes too similar to yourself provide no perturbation.
Random truncation vs. entropy truncation: entropy truncation is slightly better, but the gap is small.
Prefix probability p: p=0.2 works best; p=1.0 is slightly worse. Too many prefixes interfere with normal learning.

Why This Matters

This paper touches a deep tension in RLVR training: performance and diversity are hard to have together.

Standard RLVR is like a training system that only rewards champions — the model quickly learns the safest route but loses the ability to take unconventional ones. Prior solutions tinker with "how to reward"; this paper tinker with "where to start."

It also reveals something deeper: during exploration, correctness isn't what matters most — difference is. A wrong starting point may be more valuable than a correct one, because the correct starting point is just a road you already know, while the wrong one may lead somewhere you never imagined.

This matches human learning experience. A good math teacher doesn't only show standard answers — she deliberately shows detour solutions, even "wrong but inspiring" attempts. Students often learn more from these detours than from the shortest path.

From a technical standpoint, the engineering value is high: no extra training, no complex reward design, no algorithm changes — just prepend a prefix during data preprocessing. Nearly zero cost to mitigate a chronic RLVR problem.

Paper link: https://arxiv.org/abs/2608.27420 HTML version: https://arxiv.org/html/2608.27420v1