Summary
TIDE (arXiv:2504.20715) is the first framework for cross-architecture distillation of diffusion large language models (dLLMs). While existing dLLM distillation methods only reduce inference steps within a single architecture, TIDE transfers knowledge between models that differ in architecture, attention mechanism, and tokenizer. It introduces three modular components: TIDAL, which modulates distillation strength across training progress and diffusion timesteps to reflect the teacher's noise-dependent reliability; CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and Reverse CALM, a cross-tokenizer objective that reverses block-level likelihood matching, yielding bounded gradients and two-sided noise filtering. Using two heterogeneous pipelines, the authors distilled an 8B dense model and a 16B MoE model into 0.6B students, outperforming baselines by an average of 1.53 points across eight benchmarks and reaching a HumanEval score of 48.78 on code generation versus 32.3 for the AR baseline.
Field: NLP
Authors: Gongbo Zhang, Wen Wang, Ye Tian
Published: 2025-04-30
arXiv: 2504.20715
Abstract
Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. Existing distillation methods for dLLMs reduce inference steps within a single architecture, but none address cross-architecture knowledge transfer, where the teacher and student differ in architecture, attention mechanism, and tokenizer.
The authors present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components:
1. TIDAL — jointly modulates distillation strength across training progress and diffusion timestep, accounting for the teacher's noise-dependent reliability.
2. CompDemo — enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking.
3. Reverse CALM — a cross-tokenizer objective that reverses block-level likelihood matching, producing bounded gradients and two-sided noise filtering.
Results
Through two heterogeneous pipelines, an 8B dense model and a 16B MoE model were distilled into 0.6B student models:
- Average improvement of 1.53 points over baselines across eight benchmarks
- HumanEval score of 48.78 on code generation, far exceeding the AR baseline's 32.3
*Source:
arXiv:2504.20715*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618944