TIDE: Cross-Architecture Distillation Shrinks 8B Models into 0.6B Diffusion LLMs

Forum topic · 小凯 · 2026-04-30

Summary

This post explains TIDE (Turning the TIDE), a cross-architecture knowledge distillation framework from arXiv paper 2604.26951 by Gongbo Zhang, Wen Wang, and Ye Tian. TIDE transfers knowledge from large teacher models (8B dense and 16B MoE) into a compact 0.6B diffusion language model (dLLM), overcoming three fundamental mismatches: AR-to-diffusion architectural differences, tokenizer incompatibility, and noise-dependent teacher reliability. It introduces three components: TIDAL, a dual-schedule mechanism that adapts distillation strength across training progress and diffusion noise levels; CompDemo, a complementary mask-splitting scheme that gives the teacher visibility into masked tokens the student cannot see; and Reverse CALM, a chunk-level likelihood alignment with reverse mapping that handles differing tokenizers while keeping gradients bounded. On 8 diverse benchmarks, the 0.6B student gains an average of 1.53 points, with HumanEval code generation rising from 32.3 to 48.78 (over 50% improvement) versus an autoregressive baseline. The result points toward on-device AI: a sub-1B model that fits in a phone's memory yet approaches desktop-class code generation, suggesting knowledge can be detached from its original architecture, tokenizer, and even the autoregressive generation paradigm.

TIDE: Cross-Architecture Distillation for Diffusion Large Language Models

> Paper: Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models > Authors: Gongbo Zhang, Wen Wang, Ye Tian > arXiv: 2604.26951 > Published: 2026-04-29 > Fields: NLP / Diffusion Models / Knowledge Distillation

This is a structured summary of a long Chinese-language analysis post.

Key points

The problem: Autoregressive LLMs generate one token at a time and cannot revise earlier output. Diffusion LLMs instead denoise from random noise toward a coherent sentence — but state-of-the-art dLLMs require billions of parameters (7B–70B) to be competitive. TIDE asks whether a 0.6B "apprentice" model can learn from an 8B "master."
Why it is hard: This is not ordinary compression but cross-architecture distillation — teacher and student differ in scale (8B → 0.6B), generation paradigm (autoregressive teacher → diffusion student, or MoE teacher → dense student), and tokenizer. Standard distillation assumes teacher and student share the same architecture; TIDE must *translate* knowledge across fundamentally different architectures.

The three components of TIDE

TIDAL (Timestep- and Diffusion-aware Adaptive Learning): A dual scheduler that dynamically adjusts distillation strength along two axes — training progress (more freedom early, strict alignment late) and diffusion noise level (lower distillation weight at high noise, where the teacher itself is unreliable; higher weight at low noise). The core insight: even the teacher is not equally worth imitating at all timesteps.
CompDemo (Complementary Mask Splitting Demonstration): During training, dLLMs mask a large fraction of tokens (~60%), blinding the teacher too. CompDemo splits inputs so teacher and student receive *complementary* masked views — what the student cannot see, the teacher can. The teacher is not simply given the full answer, so the student still practices genuine completion.
Reverse CALM (Chunk-level Alignment with Likelihood Matching): Different tokenizers segment text differently, making per-token probabilities incomparable (a "unit mismatch"). Instead of aligning individual tokens, Reverse CALM matches the *joint likelihood of token chunks* and uses a *reverse* mapping (aligning the teacher's coarse chunks to the student's fine-grained chunks). Benefits: bounded gradients and two-sided noise filtering, since chunk-level matching averages out individual token fluctuations.

Experimental results

Teachers: 8B dense model and 16B MoE model.
Student: 0.6B — under 1/10 (roughly 1/26 by parameter count) of the teacher's size.
Two heterogeneous pipelines: (A) autoregressive teacher → diffusion student; (B) MoE teacher → dense student.
Evaluation: 8 diverse benchmarks spanning language understanding, reasoning, and code generation.
Results: +1.53 points on average across 8 benchmarks; HumanEval code generation: 48.78 vs. 32.3 autoregressive baseline (a >50% improvement); significant gains over existing dLLM distillation baselines.

Critical discussion (Feynman-style review in the original post)

The work avoids "cargo cult" distillation: each module targets a concrete, real mismatch rather than adding complexity for its own sake.
Open questions raised: the reverse direction (diffusion teacher → AR student) is not deeply discussed; gains on math-proving benchmarks (GSM8K, MATH) are not clearly reported; potential catastrophic forgetting of the small model's original abilities (e.g., creative writing) is not analyzed.

Implications

A 0.6B model fits in a mid-range smartphone's memory (~2–4 GB) and runs offline, enabling local, private, low-latency code assistance at roughly Copilot-level quality.
The deeper claim: LLM knowledge is not bound to the autoregressive architecture or any specific tokenizer — it can be translated into entirely different architectures, suggesting a future "one brain, many bodies" paradigm where large models explore knowledge in the cloud and small distilled models deploy it on-device.

References

1. Zhang, G., Wang, W., & Tian, Y. (2026). *Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models*. arXiv:2604.26951. 2. Sahoo, S. S., et al. (2024). *Simple and Effective Masked Diffusion Language Models*. NeurIPS 2024. 3. Gou, J., et al. (2021). *Knowledge Distillation: A Survey*. IJCV, 129, 1789–1819. 4. Hinton, G., et al. (2015). *Distilling the Knowledge in a Neural Network*. arXiv:1503.02531.

*Original post written in Chinese (~8,200 characters), analyzing arXiv paper 2604.26951.*