[论文] What Makes a Good Unmasking Policy? A Controlled Study of Learned Unma...

论文概要

研究领域: ML
作者: Jinya Sakurai, Xun Xu, Jaehong Yoon
发布时间: 2026-10-22
arXiv: 2610.31576

中文摘要

掩码扩散语言模型并行生成token，效率取决于哪些预测被提交。研究发现：低NFE下将匹配最终输出的预测标记为可提交过于宽松；隐状态单独较弱但可补充置信度；约10%的NFE减少足以抵消至多7.5M参数策略的开销。在轨迹收缩目标上训练策略，使用置信度和隐状态，生成的856K参数策略在LLaDA-8B-Instruct的GSM8K上实现最佳准确性-NFE前沿和最高吞吐量，并基本迁移到Dream-7B-Instruct。

原文摘要

Masked diffusion language models generate tokens in parallel, but efficiency depends on which predictions are committed. We find that labeling predictions matching the final output is overly permissive at low NFE, and hidden states are weak alone but can complement confidence. About a 10% NFE reduction suffices to offset policy overhead for policies up to 7.5M parameters. We train a policy on trajectory-contraction targets using confidence and hidden states. On LLaDA-8B-Instruct, the 856K-parameter policy achieves the best accuracy-NFE frontier and highest throughput on GSM8K, and largely transfers to Dream-7B-Instruct.

自动采集于 2026-10-26

#论文 #arXiv #ML #小凯