RL-Trained Models Merge Better: Fundamental Differences Between SFT and RL in Model Merging

> Paper: *Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts* > arXiv: 2607.22039 (July 24, 2025)

An Engineering Problem

Suppose you have three models: one good at math, one at coding, one at instruction following. You want to merge them into one "all-around" model.

Model merging is a popular LLM technique: instead of retraining, you average or weight-combine the parameters of multiple models to get a multi-task model. Sounds great.

But in practice, the merged model is often worse than any of the originals. The reason is "task conflict": the parameter update directions of the math model may oppose those of the code model, and merging degrades both capabilities.

This paper reports a counterintuitive finding: models trained with RL (reinforcement learning) suffer far less performance loss after merging than models trained with SFT (supervised fine-tuning). This holds regardless of merging method, RL algorithm, or base model.

Experimental Setup

The authors used three base models — Llama-3.2-3B, Llama-3.1-8B, Mistral-Small-3-24B — each fine-tuned with SFT and with three RL algorithms (PPO, GRPO, Reinforce++) on five tasks:

• Math: OpenMathInstruct-2 → GSM8K, MATH-500
• Code: OpenCodeInstruct → HumanEval, MBPP
• Instruction following: Tulu-3-SFT → IFEval, LiveBench
• Logic puzzles: Knights and Knaves
• Ranking: a ranking task

They then merged task models in pairs, triples, and quadruples, measuring post-merge performance retention.

Core Finding: RL Models Degrade Less After Merging

Across all merging methods (linear averaging, task arithmetic, TIES, etc.), RL-trained models showed significantly higher performance retention after merging than SFT-trained models.

The conclusion is robust to merging method, RL algorithm, and base model — it is not an artifact of one particular configuration.

Three Reasons Why RL Models Are More "Mergeable"

Reason 1: On-policy data constrains gradient magnitude

SFT uses fixed labeled data, and the model can "overcorrect" — making large parameter updates to minimize loss, which can overwrite knowledge from other tasks.

RL uses on-policy data: the model generates its own responses, evaluated by reward signals. It only makes small corrections where the current policy falls short. Smaller gradient updates are less likely to overwrite other tasks' abilities.

It's like studying: SFT is rote memorization of standard answers, creating strong path dependence; RL is correcting mistakes from feedback, fixing only the wrong parts while preserving what's correct.

Reason 2: RL's optimization objective is inherently gentler

SFT's loss is negative log-likelihood, which pushes the probability of the correct answer as high as possible — a "winner-take-all" objective that makes parameters charge in one direction.

RL's objective is expected return: it only requires good responses to be *more likely* than bad ones — a relatively mild goal. This mildness spreads RL's parameter updates more diffusely, reducing inter-task conflict.

Reason 3: Symmetric optimization of positive and negative samples

RL handles both positive samples (high reward) and negative samples (low reward), increasing good responses' probability while decreasing bad ones. This bidirectional adjustment makes updates more symmetric and balanced.

SFT only sees positive samples (standard answers) with no mechanism to lower bad responses' probability, making its updates one-sided.

Theoretical Analysis: Conflict Norm

The authors mathematically characterize task conflict via a "conflict norm," measuring how much two tasks' parameter update directions conflict. Theoretically, RL-trained models have a smaller conflict norm because RL's advantage function includes a baseline mechanism — subtracting a value-function estimate causes positive and negative samples' gradient updates to cancel in expectation, leaving only genuinely useful update directions. SFT lacks this cancellation, so its updates are more scattered and more likely to conflict.

Engineering Implications

1. If you plan to merge models, prefer RL training. SFT-trained models lose performance when merged; RL-trained models largely don't. 2. RL's value isn't "stronger" but "more compatible." In multi-task settings, RL's advantage lies in merge-friendliness rather than single-task performance (which is often comparable to SFT). 3. Combine SFT and RL: SFT builds foundational capability; RL serves as "de-conflicting" before merging.

Honest Assessment

• Limited experimental scale: three base models and five tasks; real-world multi-task scenarios may involve dozens of tasks with more complex conflict patterns.
• Theory relies on strong assumptions: the conflict-norm derivation assumes independent parameter updates, while updates across layers are highly correlated in practice.
• The title "Enough is as good as a feast" suggests a deeper point: RL's "gentleness" is not a defect but an advantage. In multi-task merging, you don't need to "learn the deepest" — you need to "learn the most compatible." This is inspiring, though the paper doesn't fully establish whether it applies to all scenarios.

One-Sentence Summary

RL-trained models merge better because their gradient updates are smaller in magnitude, their objective is gentler, and their optimization is more symmetric — "enough is as good as a feast" is not moral preaching, it's a mathematical theorem.

---

Paper: https://arxiv.org/abs/2607.22039 HTML version: https://arxiv.org/html/2607.22039v1 Code: not yet released (the paper used verl and OpenRLHF as training frameworks)