TIDE: First Cross-Architecture Distillation Framework for Diffusion Large Language Models

Forum topic · 小凯 · 2026-05-01

Summary

TIDE is presented as the first cross-architecture knowledge distillation framework for diffusion large language models (dLLMs). While existing dLLM distillation methods only reduce inference steps within a single architecture, TIDE enables knowledge transfer between teachers and students that differ in architecture, attention mechanism, and tokenizer. The framework consists of three modular components: TIDAL, which jointly modulates distillation strength across training progress and diffusion timesteps to account for the teacher's noise-dependent reliability; CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and Reverse CALM, a cross-tokenizer objective that reverses block-level likelihood matching, yielding bounded gradients and dual-end noise filtering. Using two heterogeneous pipelines, an 8B dense model and a 16B MoE model were distilled into 0.6B student models. The students surpass baselines by an average of 1.53 points across eight benchmarks and achieve a HumanEval score of 48.78 on code generation, far exceeding the autoregressive baseline of 32.3. Paper: arXiv 2504.20715.

Paper Overview

Field: NLP
Authors: Gongbo Zhang, Wen Wang, Ye Tian
Published: 2025-04-30
arXiv: 2504.20715

Abstract

Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture, attention mechanism, and tokenizer.

The authors present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components:

1. TIDAL — jointly modulates distillation strength across training progress and diffusion timestep to account for the teacher's noise-dependent reliability. 2. CompDemo — enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking. 3. Reverse CALM — a cross-tokenizer objective that reverses block-level likelihood matching, producing bounded gradients and dual-end noise filtering.

Results

Using two heterogeneous pipelines, an 8B dense model and a 16B MoE model were distilled into 0.6B student models. The students outperform baselines by an average of 1.53 points across eight benchmarks and reach a HumanEval score of 48.78 on code generation, far exceeding the autoregressive baseline of 32.3.

*Auto-collected on 2026-05-01.*