λ-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Predictable Budget

Forum topic · 小凯 · 2026-09-22

Summary

This paper introduces λ-Controlled GRPO, a method for stabilizing reinforcement learning fine-tuning of flow-matching image generation models with Flow-GRPO. Training is unstable in a way specific to multi-step denoising: importance ratios drift below one, become increasingly dispersed, get clipped at varying rates across denoising steps, and leave fewer usable samples late in training. Prior work treats these as separate failure modes requiring hand-tuned stabilizers. The authors show all these effects stem from a single per-step quantity called path variance, which is exactly determined by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a measurable, budgetable resource. λ-Controlled GRPO calibrates importance ratio behavior from this predictive law rather than noisy empirical statistics, and allocates gradient effort per denoising step according to predicted cost; both scales controlling the update are set by standard policy choices. On a text-to-image model under two reward settings—hard-to-render target text scored by OCR, and human preference matching scored by a preference model—λ-Controlled GRPO outperforms the strongest empirical stabilizer on both text accuracy and preference reward, while keeping late-step path variance within budget where baselines systematically overspend.

Paper Overview

Field: Machine Learning Authors: Yufeng Wang, Parivesh Priye, Meeshawn Marathe, Ramit Pahwa Published: 2026-09-18 arXiv: 2609.22041

Abstract

Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback.

Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer.

The authors show instead that all these effects arise from a single per-step quantity, called path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a measurable, budgetable resource rather than a collection of symptoms to be fixed.

λ-Controlled GRPO calibrates importance ratio behavior from this predictive law rather than noisy empirical statistics, and allocates gradient effort per denoising step according to its predicted cost. Both scales controlling the update are determined by standard policy choices, rather than introduced as free tuning parameters.

Results

On a text-to-image model under two reward settings—

Hard-to-render target text, scored by OCR
Human preference matching, scored by a preference model

—λ-Controlled GRPO outperforms the strongest empirical stabilizer on both text accuracy and preference reward. It also keeps late-step path variance within budget, exactly where baselines systematically overspend.

The result is a Flow-GRPO update calibrated by its own transition law, rather than stabilized after instability appears.

Links: arXiv:2609.22041