Decoupling Exploration from Optimization in RLVR: The ExpDis Framework

Forum topic · 小凯 · 2026-10-09

Summary

This post summarizes arXiv paper 2610.10536, which addresses a limitation of reinforcement learning with verifiable rewards (RLVR) in language models. While RLVR promises discovery of new reasoning strategies, adding strong novelty incentives to RLVR often yields limited success and can degrade model quality, and because verifiable rewards supervise only a narrow slice of model knowledge, such degradation is hard to recover from. The authors propose Exploration-Distillation (ExpDis), which decouples exploration from optimization: explorer policies are trained with a novelty bonus, their trajectories are filtered for correctness and quality, and these are distilled into a separate student policy that is then trained without novelty rewards. This cycle repeats over multiple rounds, alternating between exploration and optimization. The decoupling allows aggressive scaling of exploration without harming the student policy. On seven mathematical reasoning benchmarks across two model families, ExpDis outperforms DAPO under the same wall-clock budget, and shows improved pass@k scaling, indicating more diverse correct solutions.

Overview

Research area: NLP Authors: Saif Punjwani, Micah Goldblum Published: 2026-10-07 arXiv: 2610.10536

Abstract

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from.

Method: Exploration-Distillation (ExpDis)

Instead of mixing novelty incentives directly into RLVR, the authors decouple exploration from optimization:

1. Explore: Train one or more explorer policies with a novelty bonus added to the reward. 2. Filter: Screen the explorers' trajectories for correctness and quality. 3. Distill: Distill the filtered trajectories into an independent student policy. 4. Optimize: Train the student policy without novelty rewards.

This process is repeated over multiple rounds, alternating between exploration and optimization. Because exploration happens in separate explorer policies, it can be scaled aggressively without harming the student policy.

Results

On seven mathematical reasoning benchmarks across two model families, ExpDis outperforms DAPO under the same wall-clock budget.
ExpDis shows improved pass@k scaling, indicating the resulting models can produce more diverse correct solutions.