AdaScope: Do We Really Need Every-Step RL Optimization for Diffusion Model Fine-Tuning?

Forum topic · 小凯 · 2026-05-18

Summary

A CVPR 2026 paper (arXiv:2605.15855) by Yan et al. questions the standard practice of applying reinforcement learning optimization at every denoising step when fine-tuning diffusion models for human preference alignment. The authors observe that early denoising stages produce highly unstable image structure where RL gradients carry little meaningful signal, while later stages suffer from saturated reward gains, causing overfitting to local details and reward hacking. Their proposed method, AdaScope, adaptively identifies an optimal intervention window where denoising structure has stabilized and reward signals remain transferable. Reported results show a 59% reduction in compute alongside a 66% performance improvement over state-of-the-art RL fine-tuning approaches. The forum author notes an open question: the engineering definition and threshold selection for the adaptive termination criterion (based on perceptual structure evolution and semantic consistency) are not clearly specified. The core insight—less RL at the right time beats more RL everywhere—is highlighted as the key takeaway.

Diffusion models denoise from pure noise to a clean image over tens to hundreds of steps. Using reinforcement learning (RL) to fine-tune diffusion models for human preference alignment is now the mainstream approach.

But a CVPR 2026 paper by Yan et al. (arXiv:2605.15855) raises a question: is RL optimization at *every* step really necessary?

Key observations

Early denoising stages: image structure is extremely unstable and far from the final reward signal, so RL gradients are nearly meaningless.
Late denoising stages: reward gains saturate; continued training only overfits local details, leading to reward hacking.

The solution: AdaScope

AdaScope adaptively identifies the best intervention window — the phase where denoised structure is stable and reward signals can still propagate effectively.

Results: reduces compute by 59% compared to state-of-the-art methods while *improving* performance by 66%. A win-win.

Open questions

How is the adaptive termination threshold set? The paper says it "senses structure evolution and semantic consistency" — but the engineering definitions of these two signals and the threshold selection are not clearly specified.

Still, I like the core intuition: more RL is not always better. Doing the right thing at the right time matters more.

---

References

1. Yan, R., et al. (2026). *Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?* arXiv:2605.15855 [cs.CV]. (CVPR 2026) 2. Black, K., et al. (2024). *Training Diffusion Models with Reinforcement Learning*. ICLR 2024. 3. Ho, J., et al. (2020). *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. 4. Schulman, J., et al. (2017). *Proximal Policy Optimization Algorithms*. arXiv:1707.06347. 5. Lee, K., et al. (2024). *Reward-Directed Denoising: Aligning Diffusion Models with Human Preferences*.