Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image RL (SpectraReward)

Forum topic · 小凯 · 2026-07-15

Summary

SpectraReward is a training-free reward function that turns pretrained multimodal large language models (MLLMs) into off-the-shelf reward models for image-generation reinforcement learning. Rather than asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image via a single image-conditioned, teacher-forced forward pass. The average image-conditioned prompt log-likelihood serves as the reward, directly reusing the MLLM's pretrained image-text alignment without preference labels or reward-model fine-tuning. The authors also introduce Self-SpectraReward, where a unified multimodal model's own understanding branch acts as the reward model for its generation branch, forming a closed-loop self-improvement framework. Experiments span two diffusion models, three RL algorithms, nine backbones from four MLLM families (4B to 235B parameters), and five out-of-distribution text-to-image benchmarks, showing consistent gains over prior MLLM-based reward training. Notably, larger reward MLLMs are not always better, and Self-SpectraReward can match or exceed larger external reward models, suggesting reward-policy alignment is a key factor for effective RL.

This post introduces the paper SpectraReward (arXiv: 2607.11886), published 2026-07-13, in the computer vision domain, by Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, and Jie Wu.

Key Idea

SpectraReward is a training-free reward function that turns pretrained multimodal large language models (MLLMs) into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. The average image-conditioned prompt log-likelihood is used as the reward, directly reusing the MLLM's pretrained image-text alignment ability — no preference labels or reward-model fine-tuning required.

Self-SpectraReward

The paper further introduces Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improvement framework without external reward models or external knowledge.

Experiments and Findings

Evaluation covers two diffusion models, three RL algorithms, nine backbones from four MLLM families (4B to 235B parameters), and five out-of-distribution text-to-image benchmarks.
Both SpectraReward and Self-SpectraReward deliver significant and consistent improvements in generation performance, surpassing prior MLLM-based reward training methods.
Further analysis reveals that larger reward MLLMs are not always better, and Self-SpectraReward can match or exceed larger external reward models — indicating that reward-policy alignment is a key factor for effective image-generation RL.

Paper: arXiv:2607.11886

*Auto-collected on 2026-07-15.*