DRL: The Reward Was in Your Data All Along — Discriminator-Guided RL Fixes Flow Matching's Structural Flaw

Forum topic · 小凯 · 2026-06-20

Summary

Researchers from Meta FAIR, Columbia University, and Mila show that flow matching models systematically miss data properties — such as visual realism and structural coherence — even when training loss is low, due to a structural mismatch between the FSM training objective (computed on interpolation marginals) and the sampling distribution. Their proposed method, DRL (Discriminator-Guided RL), trains a simple linear discriminator in a frozen self-supervised representation space (e.g., DINOv2) to estimate the log density ratio between data and model samples, then uses this as a reward for KL-regularized RL via Adjoint Matching. No human preference data is required. Across SiT, JiT, REPA, and RAE, DRL consistently improves distribution alignment (SiT FID from 9.38 to 2.62; DINOv3 FD from 88.2 to 19.3) and improves scores on four held-out preference reward models. DRL also provides a better initialization for subsequent preference RL, reducing low-level statistical drift. Ablations show pretrained SSL features are essential; discriminators trained from scratch fail. The paper argues RL post-training accesses optimization directions that FSM loss provably cannot, framing flow matching as a proxy-objective misalignment problem analogous to DAgger in imitation learning.

DRL: The Reward Was in Your Data All Along — Fixing Flow Matching's Inherent Flaw with Discriminator-Guided RL

> A research team from Meta FAIR, Columbia University, and Mila found a counterintuitive fact: flow matching models miss properties present in the training data itself — such as visual realism and object structural coherence — even when training loss is low. This is not a data problem but a structural mismatch between the training objective and the sampling objective. Their solution, DRL (Discriminator-Guided RL), needs no expensive human preference data: a simple discriminator trained in a pretrained self-supervised representation space dramatically improves generation quality. SiT's FID drops from 9.38 to 2.62, DINOv3-space FD from 88.2 to 19.3, and the method generalizes across all SOTA architectures.

1. A Paradox: Why Can RL Post-Training Fix Things Flow Matching Should Have Learned From Data?

Flow matching and diffusion models are now the dominant paradigms for image/video generation. Standard pipelines have two stages:

1. Stage 1: train a base model with the Flow/Score Matching (FSM) loss 2. Stage 2: RL post-training to optimize some reward function

Stage 2 is usually motivated by "aligning with human preferences." But the team observed something strange:

RL post-training not only improves preference-related attributes, but also improves visual realism and structural coherence — attributes that should already exist in the training data.

If these properties exist in the training set, why didn't FSM learn them? Why is RL needed to "recover" them?

The paper's core claim: this is a structural flaw of the FSM loss itself, not a data or capacity problem.

2. FSM's Structural Flaw: Training on q_t, Sampling from p_t

2.1 Train-Test Distribution Mismatch

The FSM loss is computed on the interpolated marginal distribution $$q_t$$ :

$X_t = \alpha(t) X_1 + \beta(t) X_0$ , with $X_1 \sim q$ (data), $X_0 \sim \mathcal{N}(0, I)$ (noise)
Training objective: $\mathbb{E}[\|v_\theta(X_t, t) - (\dot{\alpha}(t) X_1 + \dot{\beta}(t) X_0)\|^2]$

But at sampling time, the model rolls out its own trajectory distribution $$p_t$$ . Small early errors push trajectories into low-quality regions of $$q_t$$ that training never visited, and errors compound.

Proposition 3.1 (worst case): there exist velocity fields $$v$$ with arbitrarily small FSM loss whose sampling distribution $$p$$ differs from $$q$$ on any property.

2.2 Geometric Mismatch: Velocity Space vs. Sample Space

FSM controls $\ell_2$ error in velocity space:

\|v_\theta(x, t) - v^*(x, t)\|^2

But properties we care about (e.g., "does this face look natural") are functions $$r(x)$$ in sample space. The geometries don't match.

Proposition 3.2: if $$r$$ is $$L_r$$ -Lipschitz and $$v^*$$ is $$L_v$$ -Lipschitz, then:

|\mathbb{E}_p[r] - \mathbb{E}_q[r]| \leq \varepsilon L_r \frac{e^{L_v} - 1}{L_v}

Since $$L_r$$ can be huge for visual attributes (a few pixels can turn a "hand" into a deformity), the required $\varepsilon$ may fall below the training noise floor — FSM's gradient signal is drowned out in these directions.

2.3 Experimental Validation: Distillation Cannot Copy the RL Teacher

Key experiment: train an RL teacher (Stable Diffusion 1.5, optimizing ImageReward), then train an FSM student on the teacher's samples. The student is significantly worse than the teacher in all feature spaces and fails to converge — proving RL's value lies in the optimization objective itself, not just its samples.

3. The DRL Method: Extracting the Reward From the Data Itself

3.1 Core Idea

Standard RL needs expensive, subjective preference data. DRL's insight: if we want to recover properties of $$q$$ , the optimal reward is the log density ratio between data and model:

r^*(x) = \log \frac{q(x)}{p_{base}(x)}

This can be estimated by training a discriminator — using only the original training data.

3.2 Two-Stage Algorithm

Stage 1 — Reward estimation 1. Extract representations with a frozen self-supervised encoder $\phi$ (e.g., DINOv2) 2. Train a linear discriminator $D_\psi$ to distinguish real data $$q$$ from model samples $p_{base}$ 3. Reward: $\hat{r}(x) = \text{logit}\, D_\psi(\phi(x)) = \log \frac{D}{1-D}$

Stage 2 — KL-regularized RL 1. Use Adjoint Matching (state-of-the-art RL for flow models) 2. Objective: $\max_p \mathbb{E}_{x \sim p}[\hat{r}(x)] - \frac{1}{\lambda} KL(p \| p_{base})$

3.3 Why Self-Supervised Representation Space?

Directly training a discriminator in pixel space is statistically hard and can latch onto irrelevant artifacts (e.g., JPEG traces). Constraining to a pretrained SSL space:

Lowers dimensionality, making density estimation feasible
Restricts the discriminator to semantically meaningful, human-perceptible differences

Mathematically, DRL solves:

\min_p KL(p \| p_{base}) \quad \text{s.t.} \quad p^\phi = q^\phi

The choice of SSL encoder determines which aspects DRL corrects.

4. Experimental Results

4.1 Models Tested

| Model | Architecture | Notes | |-------|-------------|-------| | SiT | Transformer | standard flow matching | | JiT | joint training | efficient training | | REPA | representation alignment | SSL features as regularization | | RAE | autoencoder | pretrained SSL as latent space |

4.2 Distribution Alignment (Fréchet Distance)

Evaluated in four feature spaces: DINOv2, DINOv3, SigLIP, InceptionV3.

Without CFG (hardest setting): DRL's tuned $\lambda$ improves all 16/16 model–feature pairs; FD often drops by more than half.
With CFG: 14/16 pairs improve.

Typical values (SiT, no CFG):

| Space | Base FD | DRL FD | Improvement | |-------|---------|--------|-------------| | DINOv2 | 159 | 58.3 | -63% | | DINOv3 | 63.7 | 29.4 | -54% | | SigLIP | 31.2 | 14.3 | -54% | | Inception | 6.43 | 2.14 | -67% |

4.3 Image Quality Without Preference Data

Evaluated with four held-out preference models — ImageReward, PickScore, Aesthetics v2.5, HPSv2 — all models improve on all of them. Since DRL never saw these rewards during training, this indicates it recovers genuine data properties rather than overfitting a specific reward.

4.4 FID (No CFG)

| Model | Base FID | DRL FID | Improvement | |-------|----------|---------|-------------| | SiT | 9.38 | 2.62 | -72% | | JiT | ~6.5 | ~3.5 | -46% | | REPA | ~4.2 | ~2.8 | -33% | | RAE | ~5.1 | ~3.2 | -37% |

4.5 DRL Provides a Better Starting Point for Preference RL

Preference RL (PRL) must simultaneously fix distributional errors and optimize subjective preferences — small $\lambda_{PRL}$ leaves structural errors unfixed; large $\lambda_{PRL}$ produces oversaturated, over-bright images. DRL + PRL consistently beats Base + PRL on HPSv2, with less drift in brightness/saturation/contrast, and coherent structures and natural colors at all $\lambda$ values.

4.6 Ablations

Discriminator architecture: linear head on frozen DINOv2 works best; MLP-2 close; fine-tuned DINOv2 slightly worse; training from scratch significantly worse — even at 95% validation accuracy, it yields useless density-ratio estimates. Pretrained SSL features are essential.
R1 regularization: crucial at $\lambda=10$ (stabilizes/smooths the reward landscape); harmful at $\lambda=1$ .
Feature space: all SSL encoders (DINOv2/v3, SigLIP) work; DINOv2-L most stable; InceptionV3 (classification-only) worst — pretraining objective matters.

5. Theoretical Insights

Distillation control: an FSM student trained on DRL-teacher samples (900k steps, >50M samples — 150× the RL data) still cannot reproduce the teacher. RL's gain comes from the optimization objective, not better data.
Calibration view: small FSM loss does not guarantee $\mathbb{E}_p[r] \approx \mathbb{E}_q[r]$ — no guarantee in the worst case, loose bounds in the regular case. RL via a Pinsker-type bound directly optimizes the quantity of interest.
Connection to imitation learning: like DAgger, flow matching suffers train-test mismatch, but worse — continuous-time ODE error compounds exponentially (the $e^{L_v}$ factor).

6. Limitations and Future Directions

1. Relies on frozen SSL features — could encoding and generation be learned jointly? 2. Finer characterization of which properties FSM struggles with in practice 3. Concrete effects of SDE vs. ODE samplers on error accumulation 4. Other on-policy objectives (e.g., MMD) and their trade-offs

7. One-Sentence Summary

Flow matching's training objective (velocity regression) and generation goal (sample quality) are structurally misaligned — like judging driving quality by steering-wheel angle. RL post-training works because it directly optimizes sample-space properties. DRL's further insight: this reward needs no human annotation — it is hiding in your raw training data, and a simple discriminator can read it out. No preferences, no architecture changes: one linear discriminator + standard RL = a qualitative leap.

---

Reference information

Paper: *The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL*
Authors: Nicolas Beltran-Velez, Felix Friedrich, Xiaofeng Zhang, Reyhane Askari-Hemmat, Xiaochuang Han, Adriana Romero-Soriano, Michal Drozdzal
Institutions: Meta FAIR, Columbia University, Mila – Québec AI Institute, McGill University, Université de Montréal
arXiv: 2606.19162
Key results: SiT FID 9.38 → 2.62 (-72%); DINOv3 FD 88.2 → 19.3 (-78%); improvements on all preference rewards across all models
Training cost: 10k discriminator steps + 3k RL steps — a small fraction of the 1M+ pretraining steps
Configuration: linear discriminator head + frozen DINOv2-L features + λ=10 + R1 gradient penalty

*Closing thought: this is not just about image generation. Any ML system where the training objective and the evaluation objective differ may suffer similar structural misalignment — perplexity vs. human evaluation in NLP, click-through rate vs. long-term user satisfaction in recommender systems. DRL's approach — extracting an evaluation-aligned reward directly from data — may be a general remedy for proxy-objective misalignment.*