Designing Transformers for Continuous Diffusion Language Models: Introducing LangDiT

Forum topic · 小凯 · 2026-10-17

Summary

A paper by Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, and Tianyang Hu (arXiv:2610.22330) presents a systematic study of Transformer architectures for continuous diffusion language models (CDLMs). The authors note that existing CDLMs typically reuse architectures from causal LLMs or image-generation Diffusion Transformers (DiTs), neither of which is optimized for CDLMs. They introduce the Language Diffusion Transformer (LangDiT), which combines feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. On OpenWebText unconditional generation of 1024 tokens, LangDiT achieves a generation perplexity (GenPPL) of 21.72 with 1024 NFE and 44.72 with 128 NFE. When adapted to a Plaid-style CDLM, it attains a leading GenPPL of 11.33. The work establishes that architecture design is a significant source of improvement for continuous diffusion language models.

Paper Overview

Research Area: Machine Learning Authors: Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, Tianyang Hu Published: 2026-10-15 arXiv: 2610.22330

Chinese Abstract (Translated)

Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly reuse transformer architectures from either causal LLMs or diffusion Transformers designed for image generation, architectures that were not optimized for CDLMs. The authors conduct a systematic study of Transformer architectures for CDLMs and introduce the Language Diffusion Transformer (LangDiT), which combines feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. In unconditional generation of 1024 tokens on OpenWebText, LangDiT achieves a GenPPL of 21.72 with 1024 NFE and 44.72 with 128 NFE. When adapted to a Plaid-style CDLM, it achieves a leading GenPPL of 11.33. These findings establish that architecture design is an important source of improvement for continuous diffusion language models.

Original Abstract

Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across diffe...

---

*Auto-collected on 2026-10-17*