RL-Trained Models Merge Better: Why SFT and RL Differ Fundamentally in Model Merging

Forum topic · ✨步子哥 · 2026-07-27

Summary

A recent arXiv paper (2607.22039) reveals a counterintuitive finding: models trained with reinforcement learning (RL) suffer far less performance degradation when merged with other models than models trained with supervised fine-tuning (SFT). Model merging combines multiple specialized models' parameters without retraining, but task conflicts—opposing parameter update directions across tasks—often degrade merged performance. The paper tested three base models (Llama-3.2-3B, Llama-3.1-8B, Mistral-Small-3-24B), five tasks (math, code, instruction following, logic puzzles, ranking), and merging methods like linear averaging, task arithmetic, and TIES, with consistent results. Three mechanisms explain RL's compatibility: (1) on-policy data keeps gradient magnitudes small, avoiding overwriting other tasks' knowledge; (2) RL's expected-return objective is gentler than SFT's winner-take-all negative log-likelihood; (3) symmetric optimization of positive and negative samples balances parameter updates. A formal 'conflict norm' analysis shows RL's advantage-function baseline cancels out unhelpful updates, yielding more focused directions. Practical implications: prefer RL training if merging is planned, and combine SFT for base capability with RL for conflict mitigation.

RL-Trained Models Are More "Mergeable": The Fundamental Difference Between SFT and RL in Model Merging

> Paper: *Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts* > arXiv: 2607.22039 (July 24, 2025)

The Engineering Problem

Suppose you have three models: one good at math, one at code, one at instruction following. You want to merge them into an all-in-one model.

Model merging is a popular LLM technique: instead of retraining, you average or weight-combine the parameters of multiple models to get a multi-task model. Sounds great.

In practice, the merged model often performs worse than any of the original models. The culprit is task conflict: the parameter update direction of the math model may oppose that of the code model, and merging drags down both capabilities.

This paper found a counterintuitive phenomenon: models trained with RL (reinforcement learning) lose far less performance after merging than models trained with SFT (supervised fine-tuning). This holds regardless of merging method, RL algorithm, or base model.

Experimental Setup

The authors trained models on five tasks using SFT and three RL algorithms (PPO, GRPO, Reinforce++) on three base models: Llama-3.2-3B, Llama-3.1-8B, and Mistral-Small-3-24B:

Math: OpenMathInstruct-2 → GSM8K, MATH-500
Code: OpenCodeInstruct → HumanEval, MBPP
Instruction following: Tulu-3-SFT → IFEval, LiveBench
Logic puzzles: Knights and Knaves
Ranking: a ranking task

They then merged models pairwise, three-way, and four-way, measuring post-merge performance retention.

Core Finding: RL Models Degrade Less After Merging

Results are clear: regardless of merging method (linear averaging, task arithmetic, TIES, etc.), RL-trained models show significantly higher performance retention after merging than SFT-trained models.

The conclusion is independent of merging method, RL algorithm, and base model—a robust pattern, not an artifact of one configuration.

Three Reasons RL Models Are More "Compatible"

Reason 1: On-policy data controls gradient magnitude

SFT uses fixed labeled data, so the model can "overcorrect"—parameters move far in one direction to minimize loss, easily overwriting other tasks' knowledge.

RL uses on-policy data: the model generates its own responses, evaluated via reward signals. This means only small corrections where the current policy is inadequate. Smaller gradient updates are less likely to overwrite other tasks.

An analogy: SFT is rote memorization of standard answers, creating strong path dependence; RL is correcting mistakes based on feedback, fixing only what's wrong while keeping what's right.

Reason 2: RL's optimization objective is inherently gentler

SFT's loss is negative log-likelihood, which maximizes the probability of correct answers—a "winner-takes-all" objective where other answers' probabilities are pushed down. This drives parameters to "charge" in one direction.

RL optimizes expected return, which doesn't require probability maximization—only that good responses are more likely than bad ones. This gentler goal makes RL's parameter updates more dispersed, reducing inter-task conflict.

Reason 3: Symmetric optimization of positive and negative samples

RL handles both positive (high-reward) and negative (low-reward) samples, simultaneously increasing good responses' probability and decreasing bad ones. This bidirectional adjustment makes updates more symmetric and balanced across directions.

SFT only uses positive samples (standard answers), with no mechanism to lower bad responses' probability, making its updates more one-sided.

Theoretical Analysis: The Conflict Norm

The authors formalize task conflict with a conflict norm, measuring how much two tasks' parameter update directions oppose each other.

Theory shows RL produces smaller conflict norms. The reason: RL's advantage function has a baseline mechanism—it subtracts a value-function estimate so positive and negative samples' gradients cancel in expectation, leaving only "truly useful" update directions. This cancellation makes RL updates more focused and less likely to conflict. SFT has no such mechanism, so its updates are more scattered.

Engineering Implications

1. If you plan to do model merging, prefer RL training. SFT-trained models degrade after merging; RL-trained models largely don't. 2. RL isn't for being "stronger"—it's for being more "mergeable." In multi-task settings, RL's value lies not in single-task performance (often comparable to SFT) but in merge-friendliness. 3. Combine SFT and RL: SFT builds base capability; RL serves as a "de-conflicting" step before merging.

Honest Assessment

Limited experimental scale: three base models, five tasks—real-world multi-task scenarios may involve dozens of tasks with more complex conflict patterns.
Theory rests on strong assumptions: the conflict norm derivation assumes independent parameter updates, though updates across layers are highly correlated in practice.
The title "Enough is as good as a feast" hints at a deeper point: RL's "gentleness" is not a flaw but an advantage. In multi-task merging, you don't need to "learn the deepest"—you need to "learn most compatibly." Insightful, though the paper doesn't fully prove it generalizes to all scenarios.

One-Sentence Summary

RL-trained models merge better because their gradient updates are smaller, their optimization objective is gentler, and their positive/negative sample optimization is more symmetric—"enough is as good as a feast" is not moralizing, it's a mathematical theorem.

---

Paper: https://arxiv.org/abs/2607.22039 HTML version: https://arxiv.org/html/2607.22039v1 Code: not yet released (training used verl and OpenRLHF)