[论文] RL Post-Training in Diffusion Transformers Is Activation Steering
研究领域: CV 作者: Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag 发布时间: 2026-11-19 arXiv: 2611.26627
论文概要
研究领域: CV 作者: Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag 发布时间: 2026-11-19 arXiv: 2611.26627
中文摘要
强化学习(RL)后训练显著提升了扩散Transformer的视觉质量、文本渲染和构图准确性。由于渲染请求文本或组合多个对象等能力本质上依赖提示,一个自然的假设是改进它们需要学习复杂的、依赖输入的变换。为隔离输入无关偏移能解释多少收益,我们用一个刻意简化的参数化探测每个RL微调模型:向冻结基础模型的块输出添加一组固定引导向量。在SD3.5-M上,这些向量的参数量比RL微调LoRA少65倍(0.29M对18.8M)。我们拟合它们以匹配公开RL微调检查点的速度预测,目标中不含奖励。尽管向量对每个提示、token和采样种子都相同,它们恢复了六个RL微调图像模型90.0-96.3%的奖励增益,以及两个RL微调视频模型89.8%和111.9%的增益。从零初始化并经在线RL训练,仅这些向量就在SD3.5-M的文本渲染(OCR)和人类偏好(PickScore)上超过了基于权重的RL的奖励增益。因此,常数偏移足以覆盖策略梯度RL后训练的大部分增益:无需新的依赖输入的变换,它便将预训练生成器引导至其已有能力可及的方向。
原文摘要
Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in...
*自动采集于 2026-11-21*
#论文 #arXiv #CV #小凯