SLAS: Super-Linear Advantage Shaping for Reward-Hacking-Resistant Post-Training of Text-to-Image Models

Forum topic · 小凯 · 2026-05-13

Summary

This paper (arXiv:2505.07245) addresses reward hacking in reinforcement-learning-based post-training of text-to-image (T2I) models. The authors observe that GRPO-style normalization can cause miscalibration, and that simply removing the prompt-level standard deviation term produces a linear advantage ascent direction that still poorly separates genuine signal from noise. They propose Super-Linear Advantage Shaping (SLAS), which extends the Fisher-Rao information metric with advantage-dependent weighting, creating a nonlinear local geometry that relaxes constraints along high-advantage directions to amplify informative updates while tightening constraints in low-advantage regions to suppress spurious gradients. Batch-level normalization stabilizes training across reward scales. Experiments show SLAS consistently outperforms the DanceGRPO baseline across multiple backbones and benchmarks, delivering faster training dynamics, improved out-of-domain performance on GenEval and UniGenBench++, and better scaling robustness, while mitigating reward hacking and preserving semantic and compositional fidelity.

Paper Overview

Field: Computer Vision Authors: Haoyuan Sun, Jing Wang, Yuxin Song Published: 2025-05-09 arXiv: 2505.07245

Background

Reinforcement-learning-based post-training, particularly Group Relative Policy Optimization (GRPO), has emerged as a robust paradigm for further advancing text-to-image (T2I) models. However, these methods are prone to reward hacking, where models exploit biases in imperfect reward functions rather than achieving genuine performance gains.

Key Contributions

The authors identify that normalization can lead to miscalibration, and that directly removing the prompt-level standard deviation term yields an optimal policy ascent direction that is linear in the advantage — but this still limits the separation of genuine signals from noise.

Method: Super-Linear Advantage Shaping (SLAS)

Revisiting functional updates from an information-geometric perspective, SLAS:

Extends the Fisher-Rao information metric with advantage-dependent weighting, introducing a nonlinear geometric structure that reshapes the local policy space.
Relaxes constraints along high-advantage directions to amplify information-rich updates.
Tightens constraints in low-advantage regions to suppress spurious gradients.
Applies batch-level normalization to stabilize training across different reward scales.

Results

Extensive evaluations show that SLAS consistently outperforms the DanceGRPO baseline across multiple backbones and benchmarks. In particular, it delivers:

Faster training dynamics
Improved out-of-domain performance on GenEval and UniGenBench++
Enhanced robustness to model scaling
Mitigated reward hacking while preserving semantic and compositional fidelity in generation

Original Abstract (excerpt)

> Recently, post-training methods based on reinforcement learning, with a particular focus on Group Relative Policy Optimization (GRPO), have emerged as the robust paradigm for further advancement of text-to-image (T2I) models. However, these methods are often prone to reward hacking, wherein models exploit biases in imperfect reward functions rather than yielding genuine performance gains. In this work, we identify that normalization could lead to miscalibration and directly removing the prompt-level standard deviation term yields an optimal policy ascent direction that is linear in the advantage but still limits the separation of genuine signals from noise. To mitigate the above issues, we propose Super-Linear Advantage Shaping (SLAS) by revisiting the functional update from an information...

---

Paper link: <https://arxiv.org/abs/2505.07245>