S2L-PO: Smaller LLMs as Natural Explorers to Break GRPO's Exploration Bottleneck via Policy-Level Diversity

Forum topic · 小凯 · 2026-06-17

Summary

A Chinese tech forum post discusses S2L-PO (Small-to-Large Policy Optimization), a reinforcement learning framework for improving GRPO training of large language models. The core insight is counterintuitive: while temperature-based token-level sampling adds noise that breaks logical coherence in reasoning chains, smaller models provide natural policy-level diversity—parameter compression creates structured, time-correlated shifts across entire reasoning strategies while preserving internal consistency. S2L-PO freezes a small model as a diversity generator, mixes its rollouts with the large model's samples in GRPO candidate groups, and progressively anneals the small model's share from 100% to 0% over the first half of training. On Qwen3 math benchmarks (Pass@1), guiding an 8B model with a 1.7B model raised AIME24 from 15.0 to 23.8 and AIME25 from 12.1 to 22.5. Ablations confirm gains come specifically from policy-level diversity: filtering it out drops performance back to the GRPO baseline. Math-trained models also improved on CommonsenseQA (63.9% to 67.8%), suggesting improved generalization. Paper: arXiv:2605.30789; code available on GitHub.

This post from zhichai.net introduces S2L-PO (*Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO*), a framework that uses small language models as structured explorers during GRPO training.

Paper: https://arxiv.org/abs/2605.30789 | Code: https://github.com/qishisuren123/S2L-PO

Key points

Counterintuitive finding: Smaller models are natural policy-level explorers. Their parameter compression creates a structured, systematic shift of the entire reasoning policy—while keeping each trajectory internally coherent. High-temperature token-level sampling, by contrast, produces diversity that breaks logical consistency mid-chain.
Token-level vs. policy-level noise:
Token-level perturbation: independent noise each step; prefix-match probability decays exponentially; cross-step signals are O(L) random walks that cancel out.
Policy-level perturbation: time-invariant parameter shift (θ̃ = θ + δ_θ); Hessian-aligned, time-correlated; O(L²) constructive interference across steps.
S2L-PO design:

Progressive annealing

α = 1 - (i-1)/(T_mix-1)

Annealing ablation: pure small-model rollouts plateau mid-training; abrupt switching causes optimization shock; progressive annealing yields the best convergence.

Main results (Qwen3, math Pass@1)

| Config | AIME24 | AIME25 | MATH-500 | OlympiadBench | |:---|:---|:---|:---|:---| | 1.7B → 8B baseline | 15.0 | 12.1 | 57.3 | 18.1 | | 1.7B → 8B S2L-PO | 23.8 | 22.5 | 61.5 | 19.7 |

Diversity scales inversely with size: the 1.7B model beats 4B/8B/14B on Self-BLEU (0.314, lower is better), Edit Diversity (0.788), and Unique Answer Ratio (0.576).
Control experiment: filtering S2L-PO's rollouts to match the large model's diversity drops AIME24 to 14.7 and AIME25 to 12.0—back to the GRPO baseline—confirming the gains come from policy-level diversity itself.
OOD generalization: on CommonsenseQA, Qwen3-8B with S2L-PO-4B reaches 67.8% vs. 63.9% for GRPO, indicating better generalization rather than domain specialization.

Implications discussed

1. "Small" is not "worse" but "different": small models excel on the policy-diversity dimension—akin to r-strategist species acting as pioneers in RL exploration. 2. Diversity is hierarchical: structured, time-correlated (policy-level) diversity is more valuable for gradient estimation than step-wise jitter, since GRPO needs distinct yet internally complete candidates. 3. Efficiency: the frozen small model can generate rollouts in parallel, be reused across runs, and costs less per sample—potentially lowering total training cost.

Open questions

Automatic tuning of the annealing schedule (T_mix = T/2 was used).
Cross-family compatibility (e.g., a Llama model guiding a Qwen model).
Validation on open-ended generation tasks beyond math.
Composability with entropy bonuses, reward shaping, and curiosity-driven exploration.

> One-line takeaway: Small models aren't a downgraded version of large models—they're their explorers. S2L-PO's cleverness is letting the large model exploit the small model's natural tendency to take "wrong" detours, which turn out to be the map for escaping local optima.

*Reference: Ren, Y., Xu, Y., Lin, Z., et al. Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO. arXiv:2605.30789.*