SpectraReward: Pretrained MLLMs as Zero-Shot Training-Free Reward Models for Image-Generation RL

Forum topic · 小凯 · 2026-07-15

Summary

SpectraReward is a training-free reward function that converts pretrained multimodal large language models (MLLMs) into off-the-shelf reward models for reinforcement learning in text-to-image generation. Rather than having the MLLM judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image via a single image-conditioned, teacher-forced forward pass, using the average image-conditioned prompt log-likelihood as the reward. This directly reuses the MLLM's pretrained image-text alignment capability, requiring no preference labels or reward-model fine-tuning. The authors also propose Self-SpectraReward, where a unified multimodal model's own understanding branch serves as the reward model for its generation branch, enabling closed-loop self-improvement without external reward models or external knowledge. Experiments span two diffusion models, three RL algorithms, nine backbones from four MLLM families (4B to 235B parameters), and five out-of-distribution text-to-image benchmarks, showing consistent gains that outperform prior MLLM-based reward training methods. Notably, larger reward MLLMs are not always better, and Self-SpectraReward can match or exceed larger external reward models, suggesting reward-policy alignment is a key factor in effective image-generation RL.

This paper introduces SpectraReward, a training-free reward function that turns pretrained multimodal large language models (MLLMs) into off-the-shelf reward models for image-generation reinforcement learning.

Key Idea

Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. The average image-conditioned prompt log-likelihood serves as the reward, directly reusing the MLLM's pretrained image-text alignment ability — with no preference labels and no reward-model fine-tuning.

Self-SpectraReward

The paper further introduces Self-SpectraReward, a special case for unified multimodal models in which the policy's own understanding branch serves as the reward model for its generation branch. This forms a closed-loop self-improvement framework requiring no external reward model or external knowledge.

Experiments and Findings

Evaluations cover two diffusion models, three RL algorithms, nine backbones from four MLLM families (4B to 235B parameters), and five out-of-distribution text-to-image benchmarks.
Both SpectraReward and Self-SpectraReward deliver significant, consistent improvements in generation performance, surpassing prior MLLM-based reward training methods.
Analysis shows that larger reward MLLMs are not always better.
Self-SpectraReward can match or exceed larger external reward models, indicating that reward-policy alignment is a key factor for effective image-generation RL.

Paper: arXiv:2607.11886