Consolidating RLVR Capabilities Across Domains: Merge vs. Mix RL vs. Multi-Teacher On-Policy Distillation

Forum topic · 小凯 · 2026-08-30

Summary

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities typically requires training separate domain experts and then consolidating them. This paper organizes three fusion paradigms by the artifacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and Multi-teacher On-Policy Distillation (MOPD) uses both. Using shared experts and data across model scales and a multi-domain benchmark suite, the authors compare all three systematically. Although average performance differs by at most 1.4 points, gaps reach 8.6 points on individual benchmarks, revealing distinct constraints: Mix RL depends on domain mixing ratios, MOPD is capped by teacher quality, and Merge compresses all expert updates into one. Practical guidance is provided for choosing the right paradigm based on deployment scenarios and constraints.

Paper Overview

Research Area: NLP Authors: Siye Wu, Kai Yang, Yuchen Cai, Xin Xu, Peng-Yuan Wang, Jiaxuan Wang, Jiashun Liu, Jiafei Lyu, Yangkun Chen, Saiyong Yang, Yanghua Xiao Published: 2026-08-27 arXiv: 2608.27409

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. The paper organizes three fusion paradigms by the artefacts they reuse:

Merge: combines expert task vectors
Mix RL: pools their datasets
MOPD (Multi-teacher On-Policy Distillation): uses both

Because these paradigms have largely been studied in isolation, how they compare and how to choose among them remained unclear. The authors compare all three using shared experts and data across model scales and a multi-domain benchmark suite.

Key Findings

Average performance differs by at most 1.4 points, but the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain characteristics.
Training dynamics expose distinct constraints:
Mix RL depends on domain mixing ratios.
MOPD is limited by the teachers.
Merge compresses all expert updates into a single one.
All three improve single-sample accuracy, but show no measurable gain or loss in solution coverage or retained capabilities.

Practical Guide

Use Merge when experts already exist and cheap consolidation matters most.
Use Mix RL when no experts are available and a unified model must be trained.
Use MOPD when preserving domain-specific gains matters more than surpassing the teachers.