λ-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource (arXiv 2609.22041)

Forum topic · 小凯 · 2026-09-22

Summary

This paper introduces λ-Controlled GRPO, a method for stabilizing reinforcement learning in flow-matching image generation models. While Flow-GRPO enables reward-based optimization of denoising samplers, its training suffers from instability: importance ratios drift below one, become increasingly dispersed, and get clipped at varying rates across denoising steps, leaving fewer usable samples late in training. The authors show these effects stem from a single per-step quantity, path variance, exactly determined by the sampler's Gaussian transition kernel and cheaply estimable during training. Rather than treating instability symptoms with separate hand-tuned stabilizers, they reframe it as a measurable, budgetable resource. λ-Controlled GRPO calibrates importance ratio behavior from this predictive law and allocates gradient effort per denoising step according to predicted cost. On a text-to-image model under two reward settings (OCR-scored text rendering and preference-model-scored human preference), it outperforms the strongest empirical stabilizers in text accuracy and preference reward while keeping late-step path variance within budget. Paper: arXiv 2609.22041, published 2026-09-18 by Yufeng Wang, Parivesh Priye, Meeshawn Marathe, and Ramit Pahwa.

Paper Overview

Field: Machine Learning
Authors: Yufeng Wang, Parivesh Priye, Meeshawn Marathe, Ramit Pahwa
Published: 2026-09-18
arXiv: 2609.22041

Key Points

Reinforcement learning is increasingly used to align image generators with reward signals. Flow-GRPO extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy optimized from reward feedback.
Training in this setting is unstable in a way specific to multi-step denoising: policy updates change systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training.
Prior work treats these effects as separate failure modes, each addressed with a hand-tuned stabilizer. This paper shows they all arise from a single per-step quantity the authors call path variance.
Path variance is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a measurable, budgetable resource rather than a pile of symptoms to fix.
λ-Controlled GRPO calibrates importance ratio behavior from this predictive law (instead of noisy empirical statistics) and allocates gradient effort per denoising step according to its predicted cost. The two scales controlling updates are set by standard policy choices, not introduced as free hyperparameters.
On a text-to-image model under two reward settings — hard-to-render target text scored by OCR, and human preference matching scored by a preference model — λ-Controlled GRPO outperforms the strongest empirical stabilizers on both text accuracy and preference reward.
It also keeps path variance in late steps within budget, where baselines systematically overspend, yielding a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.

Abstract (Original)

> Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance.

*Auto-collected on 2026-09-22*