Why RL-Trained Models Merge Better Than SFT Models

> Paper: *Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts* > arXiv: 2607.22039 > HTML: https://arxiv.org/abs/2607.22039v1

The Engineering Problem

Suppose you have three fine-tuned models: one for math, one for code, and one for instruction following. You want to merge them into a single generalist.

Model merging is a popular LLM technique. It averages or combines parameters from multiple models to produce a multi-task model without retraining. In practice, however, merged models often perform below each parent model because of task conflict: parameter updates for math may point in the opposite direction from updates for code, and averaging cancels out both capabilities.

The paper reports a counterintuitive result: models fine-tuned with reinforcement learning lose far less performance after merging than models fine-tuned with supervised fine-tuning (SFT). This holds across merging methods, RL algorithms, and base models.

Experimental Setup

The authors used three base models — Llama-3.2-3B, Llama-3.1-8B, and Mistral-Small-3-24B — and trained each on five tasks with both SFT and three RL algorithms (PPO, GRPO, Reinforce++):

• Math: OpenMathInstruct-2 → GSM8K, MATH-500
• Code: OpenCodeInstruct → HumanEval, MBPP
• Instruction following: Tulu-3-SFT → IFEval, LiveBench
• Logic puzzles: Knights and Knaves
• Ranking

They then merged pairs, triples, and quadruples of task models and measured performance retention.

Core Finding

Across linear averaging, task arithmetic, and TIES merging, RL-trained models retained significantly more performance than SFT-trained models. The result is robust: it does not depend on the merging method, the RL algorithm, or the base model.

Why RL Models Are More "Merge-Friendly"

Reason 1: On-Policy Data Controls Gradient Magnitude

SFT uses a fixed labeled dataset. To minimize loss, parameters update aggressively in one direction, and large updates can overwrite knowledge from other tasks.

RL uses on-policy data: the model generates outputs and receives a reward signal. Updates are small corrections applied only where the current policy is weak, which preserves other capabilities.

Reason 2: A Softer Optimization Objective

SFT's negative log-likelihood loss is a winner-take-all objective: it pushes the probability mass onto the correct answer as high as possible while pushing alternatives as low as possible.

RL optimizes expected return, which only requires that good answers be more probable than bad ones. This relative objective produces more distributed updates that do not concentrate along a single direction, reducing inter-task conflict.

Reason 3: Symmetric Optimization of Positive and Negative Samples

RL simultaneously increases the probability of high-reward outputs and decreases the probability of low-reward outputs. This bidirectional adjustment keeps parameter updates balanced across directions.

SFT only optimizes positive samples (the reference answer) and lacks a mechanism for reducing the probability of bad outputs, leading to more one-sided updates.

Theoretical Analysis: The Conflict Norm

The authors formalize task conflict with a conflict norm that measures how much two tasks' parameter updates clash. Theoretical analysis shows RL-trained updates have a smaller conflict norm because the advantage function subtracts a value-function baseline: positive and negative gradients cancel in expectation, leaving only the truly informative update component.

SFT lacks this cancellation, producing updates that spread more directions and clash more often with other tasks.

Engineering Implications

1. If you plan to merge models, prefer RL-fine-tuned checkpoints. 2. In multi-task settings, RL's value lies not in higher single-task scores but in merge friendliness — RL-trained models often match SFT on single tasks but lose far less when merged. 3. Combine SFT and RL: use SFT to build core capabilities, then RL to "de-conflict" before merging.

Honest Assessment

• The experiments cover only three bases and five tasks. Real multi-task scenarios can involve dozens of tasks with more complex conflicts.
• The conflict-norm derivation assumes parameter updates are independent, but updates across layers are highly correlated in practice.
• The title "Enough is as good as a feast" suggests a deeper view: RL's "softness" is a feature, not a bug. The paper hints at this but does not fully prove it applies to every scenario.

One-Sentence Summary

RL-trained models merge more cleanly because their gradients are smaller, their objective is softer, and their positive/negative updates are symmetric — "enough is as good as a feast" is a mathematical theorem, not a moral slogan.

---

Paper: https://arxiv.org/abs/2607.22039 HTML: https://arxiv.org/html/2607.22039v1 Training frameworks used: verl, OpenRLHF

FAQ

Q1: Who is this article for? Practitioners, researchers, and students working in AI, machine learning, and deep learning.

Q2: What are the key takeaways?

• An engineering problem (task conflict in merging)
• Experimental setup across three bases and five tasks
• Core finding: RL models retain more performance after merging

Q3: Is the code open-sourced? The paper itself does not release code, but it builds on the verl and OpenRLHF training frameworks.