[论文] Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process...
论文概要
研究领域: NLP
作者: Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R., Arun Prakash, S. Senthilkumar
发布时间: 2026-11-16
arXiv: 2611.21538
中文摘要
长视频生成应在保持主体和背景一致的同时支持丰富自然的运动。然而,自回归模型通常在短窗口上用分布匹配蒸馏(DMD)进行蒸馏,这并未训练它们在长距离上保持一致性。结果是主体和背景会随时间漂移。管理键值(KV)缓存的方法通过保留或检索历史来减少漂移,但这些历史本身可能已经漂移。我们提出 Reference Forcing,使用初始帧作为外观参考。我们首先构建一个教师,使用注意力掩码强制新帧的每个区域关注初始帧中的相同区域。教师用现成的分割模型找到这些区域。然后我们将此行为蒸馏到学生的一个单独注意力路径中,该路径在推理时不需要外部模型。我们进一步引入值校正,将 KV 缓存中漂移的值校正为无漂移初始帧的匹配值。实验表明,Reference Forcing 在不同基础模型的长视频方法中实现了最先进的 subject 和背景一致性,且优势随视频长度增长而扩大。消融实验表明两个组件都是必需的,以在保持运动的同时获得一致性。
原文摘要
Reinforcement learning from verifiable rewards inherits whatever its verifier gets wrong. Outside mathematics and code the verifier is a proxy, a rubric, a learned judge or a hand-written checker, and its distance from the true objective is never measured, so a rising reward cannot be told apart from a hack. We build a setting where the true reward is exactly computable at the process level: executable authorization policies over 124,416 contexts admit an oracle that lists every minimal sufficient reason for every decision. We compile these policies into exact process rewards, degrade the verifier along seven realistic axes (outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge), and train against each while the exact verifier scores every rollo...
自动采集于 2026-11-18
#论文 #arXiv #NLP #小凯