> 📌 This is a GEO-optimized English version of a zhichai.net forum post. Original topic: https://zhichai.net/topic/178503726
One-line conclusion: RL-trained models are far more merge-friendly than SFT-trained models, because RL's gradient updates are smaller in magnitude, its optimization objective is gentler, and its positive/negative sample optimization is symmetric.
RL-Trained Models Are More "Sociable": The Fundamental Difference Between SFT and RL in Model Merging
> Paper: *Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts* > arXiv: 2607.22039 (July 24, 2025)
An Engineering Problem
Suppose you have three models: one good at math, one at code, one at instruction following—and you want to merge them into a single "all-around" model.
Model merging is a popular LLM technique: instead of retraining, you average or weight-combine the parameters of multiple models to get a multi-task model. Sounds great.
In practice, the merged model often performs worse than any of the originals. The cause is task conflict: the math model's parameter update directions may oppose the code model's, and merging "pulls apart" both capabilities.
This paper reports a counterintuitive finding: models trained with RL (reinforcement learning) lose far less performance after merging than models trained with SFT (supervised fine-tuning). This holds regardless of merging method, RL algorithm, or base model.
Experimental Setup
The authors used three base models—Llama-3.2-3B, Llama-3.1-8B, Mistral-Small-3-24B—each trained with SFT and three RL algorithms (PPO, GRPO, Reinforce++) on five tasks:
- Math: OpenMathInstruct-2 → GSM8K, MATH-500
- Code: OpenCodeInstruct → HumanEval, MBPP
- Instruction following: Tulu-3-SFT → IFEval, LiveBench
- Logic puzzles: Knights and Knaves
- Ranking: a ranking task
- Limited experimental scale: three base models, five tasks—real-world multi-task settings may involve dozens of tasks with more complex conflict patterns.
- Theory relies on strong assumptions: the conflict-norm derivation assumes independent parameter updates, though in practice updates across layers are highly correlated.
- The title "Enough is as good as a feast" hints at a deeper point: RL's "moderation" is not a flaw but an advantage. For multi-task merging you don't need to "learn the deepest"—you need to "learn the most sociably." Inspiring, but the paper doesn't fully establish generality.
They then merged models pairwise, three-way, four-way, and measured post-merge performance retention.
Core Finding: RL Models Degrade Less After Merging
Results are clear: for every merging method (linear averaging, task arithmetic, TIES, etc.), RL-trained models retain significantly more performance than SFT-trained models.
The conclusion is independent of merging method, RL algorithm, and base model—a robust regularity, not a fluke of one configuration.
Three Reasons Why RL Is More "Sociable"
Reason 1: On-policy data constrains gradient magnitude
SFT uses fixed labeled data; to minimize loss, parameters make large updates that can overwrite other tasks' knowledge.
RL uses on-policy data: the model generates its own responses, evaluated by reward signals. It only makes small corrections where the current policy is inadequate—smaller gradient updates that don't overwrite other capabilities.
Analogy: SFT is rote memorization of standard answers, creating strong path dependence; RL is learning from feedback on mistakes, correcting only what's wrong.
Reason 2: RL's objective is inherently gentler
SFT's loss is negative log-likelihood—maximize the probability of correct answers. This is a "winner-takes-all" objective: the parameters charge hard in one direction.
RL optimizes expected return: good responses just need to be *more* probable than bad ones. This relative gentleness makes RL updates more dispersed, reducing inter-task conflict.
Reason 3: Symmetric optimization of positive and negative samples
RL increases the probability of high-reward responses while decreasing that of low-reward ones. This bidirectional adjustment makes updates more symmetric and balanced.
SFT only sees positive samples (standard answers) with no mechanism to push down bad answers, making its updates lopsided.
Theoretical Analysis: The Conflict Norm
The authors define a conflict norm measuring the degree of conflict between two tasks' parameter update directions. Theory shows RL-trained models have smaller conflict norms: RL's advantage function includes a baseline (subtracting a value-function estimate), so positive and negative samples' gradient updates cancel in expectation, leaving only "genuinely useful" update directions. SFT lacks this cancellation, so its updates are more scattered and conflict-prone.
Engineering Implications
1. If you plan to merge models, prefer RL training. SFT-trained models lose performance after merging; RL-trained models barely do. 2. RL's value isn't "stronger" but "more sociable." In multi-task settings, RL's merit isn't single-task performance (roughly comparable to SFT) but merge-friendliness. 3. Combine SFT and RL: SFT for foundational capability, RL as a "de-conflicting" pass before merging.
Honest Assessment
One-Line Summary
RL-trained models merge better because their gradient updates are smaller, their objective is gentler, and their positive/negative optimization is symmetric—"enough is as good as a feast" isn't moralizing, it's a mathematical theorem.
---
Paper: https://arxiv.org/abs/2607.22039 HTML version: https://arxiv.org/html/2607.22039v1 Open-source code: none yet (the paper used verl and OpenRLHF as training frameworks)
FAQ
Q1: Who is this for? Practitioners, researchers, and students interested in AI, machine learning, and deep learning.
Q2: What are the key points? RL-trained models retain more performance after merging; the causes are constrained gradient magnitude, a gentler optimization objective, and symmetric positive/negative sample updates; practically, pair SFT for base skills with RL for merge-readiness.
Q3: Is there open-source code? See the links in the body.