[论文] Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Mo...

论文概要

研究领域: NLP 作者: Seo Hyun Kim, Sunwoo Hong, Younwoo Choi, Chen-Hao Chao, Se-Young Yun, Rahul G. Krishnan 发布时间: 2026-10-02 arXiv: 2610.03665

中文摘要

掩码扩散语言模型（dLM）为复杂推理提供了有前景的并行替代方案。但它们面临独特的信用分配挑战：去噪过程中的少数关键承诺会急剧降低剩余掩码位置的不确定性，从而塑造大部分响应。大多数 dLM 的后训练方案不利用这一信号来决定训练哪些 token：它们通常训练最终文本或将奖励分配给整个去噪步骤，而非选择塑造响应的个别承诺。我们提出 Pivot-SD——一个高效的离线自蒸馏框架，仅监督这些高影响的承诺（pivot）。Pivot-SD 使用信息增益度量来选择 pivot，衡量对剩余掩码位置的不确定性缩减。成功轨迹中的 pivot 用交叉熵训练，失败轨迹中的 pivot 用有针对性的 unlikelihood 训练，其余部分保持不动。仅使用 200 个问题和每个问题四次采样，Pivot-SD 在数学和代码基准上使 LLaDA-8B-Instruct 超越了全序列 SFT 和预算匹配的扩散 RL 基线。

原文摘要

Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sharply reduce the uncertainty over the remaining masked positions and shape much of the response. Most post-training recipes for dLMs do not use this signal to decide which tokens to train on: they typically train on the final text or assign rewards to whole denoising steps, rather than selecting the individual commitments that shape the response. We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments (pivots). Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction ...

#论文 #arXiv #NLP #小凯