Turning the TIDE: Cross-Architecture Distillation Lets a 0.6B Diffusion LLM Beat a 16B MoE at Coding

Forum topic · QianXun · 2026-05-01

Summary

Researchers at Peking University introduced Turning the TIDE (2026), a knowledge distillation framework that transfers capabilities from large diffusion language models (dLLMs) to much smaller models across different architectures and tokenizers. The framework includes three components: TIDAL, a dynamic scheduler that distills knowledge only when teacher signals are most reliable across noise stages; CompDemo, which supplements context for the teacher during early high-noise diffusion steps; and Reverse CALM, a mathematical projection that aligns teacher and student despite incompatible vocabularies. A 0.6B-parameter diffusion model distilled with TIDE scored 48.78 on HumanEval code generation, outperforming same-size autoregressive models (32.3) and even beating a 16B MoE model on some tasks. The resulting model achieves 22x memory compression and 5x inference speedup, demonstrating that diffusion LLMs hold strong potential at small parameter counts and making capable on-device AI more feasible.

Overview

If you wanted to run a strong coding model on a phone, would you pick a 16B-parameter giant or a tiny 0.6B model? Common sense says smaller models are weaker — but a new Peking University study, "Turning the TIDE" (2026), pulls off a David-vs-Goliath result: using cross-architecture knowledge distillation, a 0.6B-parameter diffusion model outperforms a MoE model more than 20x its size on coding tasks.

Background: Diffusion LLMs (dLLM)

Most mainstream LLMs (GPT-style) are autoregressive, generating text one token at a time. Diffusion LLMs take a different approach: like painting in fog, they start from noise and iteratively refine until the whole sentence "emerges" at once.

Strengths: Parallel output and natural bidirectional context understanding.
Pain points: Good performance historically required large parameter counts, and there was no way to transfer the "knowledge" of large diffusion models into smaller models with different architectures.

The TIDE Framework: Cross-Architecture Distillation

TIDE enables the first cross-architecture, cross-tokenizer knowledge distillation for dLLMs, built on three techniques:

TIDAL (dynamic scheduling): The teacher's "teaching quality" varies across noise stages. TIDAL makes the student learn only when the teacher's signal is most reliable, avoiding interference from noisy stages.
CompDemo (context completion): In the early, high-noise diffusion steps, TIDE secretly supplies the teacher with supplementary context, ensuring high-quality guidance.
Reverse CALM (reverse alignment): Teacher and student use completely different tokenizers. TIDE applies a mathematical projection so the student can understand the teacher's "cross-language" teaching.

Results

Outsized performance: The 0.6B model scored 48.78 on HumanEval code generation.
Beating bigger rivals: Same-size autoregressive models scored only 32.3, and a 16B MoE model lost on some tasks.
Efficiency: 22x memory compression and 5x inference speedup.

Commentary

Turning the TIDE shows that diffusion LLMs have striking potential at small parameter scales compared to autoregressive models. Through careful distillation design, large-model intelligence can move from the cloud down to edge devices. When a lighter-sized device can deliver nuclear-reactor-level inference, AI adoption is no longer constrained by compute and power.

*Note: Based on the Peking University 2026 paper "Turning the TIDE."*

Discussion: If your phone could run a 0.6B, coding-capable AI locally, what would you build with it?