Paper Overview
Research Area: NLP Authors: Keya Hu, Linlu Qiu, Yiyang Lu Published: 2025-05-09 arXiv: 2505.07246
Summary
Diffusion and flow-based models have become the de facto approaches for generating continuous data, such as images and videos, and their success has inspired growing interest in applying them to language modeling. However, unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens.
This paper demonstrates that continuous DLMs can be made effective with minimal adaptation to the discrete domain. The authors propose Embedded Language Flows (ELF), a class of diffusion models operating in continuous embedding space based on continuous-time Flow Matching.
Key Ideas
- Continuous embedding space: Unlike existing DLMs, ELF predominantly stays within the continuous embedding space throughout generation.
- Late discretization: Only at the final time step does ELF map its continuous representations to discrete tokens, using a shared-weight network.
- Technique transfer: This formulation makes it straightforward to migrate well-established techniques from image-domain diffusion models, such as classifier-free guidance (CFG).
- arXiv: https://arxiv.org/abs/2505.07246
Results
Experiments show that ELF significantly outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results indicate that ELF offers a promising path toward effective continuous diffusion language models.