ParetoSlider: Diffusion Post-Training Framework for Continuous Pareto Front Control

Forum topic · 小凯 · 2026-04-24

Summary

ParetoSlider is a multi-objective reinforcement learning (MORL) post-training framework for diffusion models, introduced in an arXiv paper (2604.20816) by Shelly Golan, Michael Finkelson, and Ariel Bereslavsky. Most RL post-training methods collapse multiple rewards into a fixed weighted sum, locking the model into a single trade-off point at training time. ParetoSlider instead trains a single diffusion model to approximate the entire Pareto front by using continuously varying preference weights as a conditioning signal. This lets users navigate optimal trade-offs between conflicting goals—such as prompt adherence versus source fidelity in image editing—at inference time, without retraining or maintaining multiple checkpoints. The authors evaluate the approach on three state-of-the-art flow-matching backbones (SD3.5, FluxKontext, and LTX-2), showing that the single preference-conditioned model matches or exceeds baselines trained for fixed reward trade-offs while uniquely offering fine-grained control over competing generation objectives.

Paper Overview

Field: Computer Vision (CV)
Authors: Shelly Golan, Michael Finkelson, Ariel Bereslavsky
Published: 2026-04-22
arXiv: 2604.20816

Summary

Reinforcement learning (RL) post-training has become the standard method for aligning generative models with human preferences, but most approaches rely on a single scalar reward. When multiple criteria matter, the prevailing practice of "early scalarization"—collapsing rewards into a fixed weighted sum—commits the model to a single trade-off point at training time. This provides no inference-time control over inherently conflicting goals, such as prompt adherence versus source fidelity in image editing.

The authors introduce ParetoSlider, a multi-objective reinforcement learning (MORL) framework that trains a single diffusion model to approximate the entire Pareto front. By training the model with continuously varying preference weights as a conditioning signal, users can navigate optimal trade-offs at inference time without retraining or maintaining multiple checkpoints.

Key Findings

ParetoSlider was evaluated on three state-of-the-art flow-matching backbones: SD3.5, FluxKontext, and LTX-2.
A single preference-conditioned model matches or exceeds the performance of baselines separately trained for fixed reward trade-offs.
The framework uniquely provides fine-grained control over competing generation objectives at inference time.

Original Abstract (excerpt)

> Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences, yet most methods rely on a single scalar reward. When multiple criteria matter, the prevailing practice of early scalarization collapses rewards into a fixed weighted sum. This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals -- such as prompt adherence versus source fidelity in image editing. We introduce ParetoSlider, a multi-objective RL (MORL) framework that trains a single diffusion model to approximate the entire Pareto front...

---

*Auto-collected on 2026-04-24.*

ParetoSlider: Diffusion Post-Training Framework for Continuous Pareto Front Control

Summary

Paper Overview

Summary

Key Findings

Original Abstract (excerpt)

Tags