Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms (Merge, Mix RL, MOPD)

Paper Overview

• Field: NLP
• Authors: Siye Wu, Kai Yang, Yuchen Cai, Xin Xu, Peng-Yuan Wang, Jiaxuan Wang, Jiashun Liu, Jiafei Lyu, Yangkun Chen, Saiyong Yang, Yanghua Xiao
• Published: 2026-08-27
• arXiv: 2608.27409

Summary

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. This paper organizes three fusion paradigms by the artefacts they reuse:

• Merge combines expert task vectors
• Mix RL pools their datasets into a single RL training run
• Multi-teacher on-policy distillation (MOPD) uses both experts and data

Because these approaches have largely been studied in isolation, how they compare and how to choose among them has remained unclear. The authors compare all three using shared experts and data across model scales and a multi-domain benchmark suite.

Key Findings

• Although average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain interaction effects.
• Training dynamics expose distinct constraints:
• Mix RL depends on the domain mixing ratio.
• MOPD is limited by the teachers.
• Merge compresses all expert updates into one.
• All three approaches improve single-sample accuracy, but show no measurable gains or losses in solution coverage or retained capabilities.

Practical Guide

• Merge: when experts already exist and cheap fusion matters most.
• Mix RL: when training a unified model without pre-trained experts.
• MOPD: when preserving domain-specific gains matters more than surpassing the teachers.

*Auto-collected on 2026-08-30.*