[论文] Turning the TIDE: Cross-Architecture Distillation for Diffusion Large ...

论文概要

研究领域: NLP
作者: Gongbo Zhang, Wen Wang, Ye Tian
发布时间: 2025-04-30
arXiv: 2504.20715

中文摘要

扩散大语言模型（dLLMs）支持并行解码和双向上下文，但最先进的dLLMs需要数十亿参数才能达到有竞争力的性能。虽然现有的dLLM蒸馏方法可以在单一架构内减少推理步骤，但都无法解决跨架构知识迁移问题——即教师模型和学生模型在架构、注意力机制和分词器上均不相同。我们提出TIDE，首个面向dLLM的跨架构蒸馏框架，包含三个模块化组件：（1）TIDAL，联合调节训练进度和扩散步长上的蒸馏强度，以适应教师模型噪声依赖的可靠性变化；（2）CompDemo，通过互补掩码分割丰富教师模型的上下文，改善重度掩码下的预测；（3）Reverse CALM，一种跨分词器目标函数，反转块级似然匹配，产生有界梯度和双端噪声过滤。通过两条异构流水线将8B稠密模型和16B MoE模型蒸馏到0.6B学生模型，在八个基准上平均超越基线1.53分，在代码生成上HumanEval得分达到48.78，远超AR基线的32.3。

原文摘要

Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture, attention mechanism, and tokenizer. We present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components: (1) TIDAL, which jointly modulates distillation strength across training progress and diffusion timestep to account for the teacher's noise-dependent reliability; (2) CompDemo, which enriches the teacher's context via complementary mask splitting to improve pred...

自动采集于 2026-05-01

#论文 #arXiv #NLP #小凯