Weak Models Coaching Strong Models: A Counterintuitive Strategy Against Entropy Collapse in RLVR Training

A Strange Experiment

Imagine training a math prodigy (Qwen2.5-7B) on competition problems. The standard approach is reinforcement learning—reward correct answers, penalize wrong ones. After a few hundred training steps, her single-attempt accuracy jumps from 27.8% to 38.3%.

But there's a side effect: she increasingly solves problems using only one method. If you let her try 128 times, coverage drops from 72.2% before training to 67.8% after. Training makes her stronger, but also narrower.

This is the "entropy collapse" problem in RLVR (Reinforcement Learning with Verifiable Rewards): the model converges early onto a few high-reward reasoning paths and never explores other possibilities again.

Researchers now propose a counterintuitive fix: have a weaker small model (e.g., Gemma-2-2B) generate a few reasoning steps, then feed those steps as an "opening" to the large model and let it continue from there.

It sounds absurd—letting the struggling student coach the star pupil? Yet the result: pass@128 rises from 67.8% to 70.7% while pass@1 stays nearly unchanged. Even more surprising: most of these small-model "openings" are wrong.

The paper is arXiv:2608.27420, "Boosting LLM Exploration via Weak-Model Guidance in RLVR."

Entropy Collapse: Why "Getting Stronger" and "Getting Narrower" Are Twins

To understand the problem, first consider how RLVR trains a model.

Given a math problem \(q\), the model generates a reasoning trajectory \(r\), a verifier checks the answer, and a reward \(R(q, r)\) is assigned. The training objective maximizes expected reward—essentially telling the model: "walk the high-scoring paths more often."

The problem: once the model finds a path that reliably scores, it reinforces it. Probability mass rapidly concentrates on that path while others are pushed toward zero. From an information-theoretic view, the policy's entropy is collapsing.

The cost of entropy collapse is loss of diversity. The model becomes a one-trick player—strong at that trick, but stuck whenever it fails. pass@1 improves, but pass@k (probability of at least one correct answer in k attempts) drops at large k.

Prior solutions work from inside the algorithm: entropy regularization, KL coefficient tuning, reward shaping. This paper takes a different angle—injecting perturbation from outside the model.

Cross-Model Prefixes: Letting a "Stranger" Start the Solution

The core method, Prefix-Completion RLVR, is extremely simple:

1. Pick a small model from a different family than the target model (e.g., LLaMA-3.2-1B or Gemma-2-2B) 2. Have it generate reasoning for the problem, then randomly truncate it at some step to form a "prefix" 3. Append this prefix after the problem and let the large model continue 4. Train the large model with standard GRPO

Formally, standard RLVR samples a full solution from scratch: \(r \sim \pi_\theta(\cdot | q)\). Prefix-Completion samples a continuation given the prefix \(\tilde{r}\): \(r_{\text{suf}} \sim \pi_\theta(\cdot | q, \tilde{r})\).

A probability \(p\) controls how often prefixes are injected during training. \(p=0\) is standard GRPO, \(p=1\) means every rollout has a prefix. The paper finds \(p=0.2\) is a good balance.

There's also a detail called Entropy-Based Prefix Truncation: instead of truncating at a random step L, look at the small model's entropy at each step. Truncate where entropy is low (where the small model is confident), preserving the part it's "sure" of—even if that confidence is wrong.

Why It Works: Perturbation Matters More Than Correctness

This is the paper's most counterintuitive finding.

The researchers used DeepSeek-V4-Flash to evaluate prefix quality, categorizing prefixes into four types:

• (a) No substantive guidance: the prefix offers no useful reasoning direction
• (b) Completely wrong and misleading: the prefix steers the model down a wrong path
• (c) Flawed but acceptable: partially useful despite errors
• (d) Fully correct and appropriate: correct reasoning

The result is striking: most prefixes fall into (a) or (b)—useless or flat-out wrong. For prefixes from LLaMA-3.2-1B, "completely wrong and misleading" exceeds 50%.

But it works anyway. Why?

The answer lies in the nature of exploration. Exploration doesn't need "correct guidance"—it needs "different starting points." Because the small model comes from a different family with far fewer parameters, its reasoning style and distribution differ sharply from the large model's. This distributional discrepancy forces the large model to continue reasoning from an unfamiliar state—even if that state is "wrong," it opens a path the large model would never have taken on its own.

The paper's training dynamics analysis confirms this:

• Policy entropy: prefix-trained models maintain notably higher entropy early on, delaying collapse
• Average reward: prefix-trained models earn lower average reward—the "cost of exploration," since prefixes often mislead
• Zero-reward ratio: prefix-trained models produce more zero-reward samples—attempting more "hard" paths
• Non-trivial sample ratio: prefix-trained models have more samples that are neither fully correct nor fully wrong—exploring the boundary region

It's like sending a chess master to a tournament where an amateur plays the opening moves of every game. The openings may be terrible, but the master is forced to find her way out of unfamiliar positions—and that trains her adaptability. The value of exploration lies not in how good the guide is, but in how new the road is.

The Numbers

Main experiments use Qwen2.5-7B and Qwen2.5-Math-7B as targets, with LLaMA-3.2-1B and Gemma-2-2B as small models, evaluated on 6 math reasoning benchmarks (AIME 2024/2025, AMC 2023, MATH 500, Minerva, Olympiad Bench).

Key results (Qwen2.5-7B):

| Method | pass@1 | pass@64 | pass@128 | |--------|--------|---------|----------| | Base model | 27.83 | 67.98 | 72.15 | | + GRPO | 38.29 | 65.02 | 67.76 | | + Gemma-2-2B prefix | 39.01 | 66.78 | 70.71 |

Note pass@128: standard GRPO training drops it from 72.15 to 67.76 (entropy collapse), while prefixes restore it to 70.71. pass@1 also slightly improves.

Ablations are also interesting:

• Same-family prefixes: using Qwen2.5-1.5B or Qwen2.5-7B itself to generate prefixes works clearly worse than cross-family small models. This confirms "distributional discrepancy" is key—prefixes too similar to the model itself provide no perturbation.
• Random truncation vs. entropy truncation: entropy truncation is slightly better, but the gap is small.
• Prefix probability p: p=0.2 works best; p=1.0 is slightly worse. Too many prefixes interfere with normal learning.

Why This Matters

This paper touches a deep tension in RLVR training: performance and diversity are hard to reconcile.

Standard RLVR is like a training system that only rewards champions—the model quickly learns the safest path but loses the ability to take unconventional ones. Previous solutions tinkered with "how to reward"; this paper tinkers with "where to start."

It also reveals a deeper truth: during exploration, correctness matters less than difference. A wrong starting point can be more valuable than a correct one—because the correct starting point is a road you already know, while a wrong one may lead somewhere you never imagined.

This matches human learning. A good math teacher doesn't only show standard solutions; she deliberately demonstrates "winding" approaches and even "wrong but inspiring" attempts. Students often learn more from these detours than from the shortest path.

Technically, the method's engineering value is high: no extra training, no complex reward design, no algorithm changes—just prepend a prefix during data preprocessing. Nearly zero cost to alleviate a chronic RLVR problem.

Paper link: https://arxiv.org/abs/2608.27420 HTML version: https://arxiv.org/html/2608.27420v1