Summary
ConvergeFlow (arXiv:2508.17626, by Na Li, Yuchen Jiao, and Changxiao Cai) is a new embedding-space flow-based language model that removes the need for a cross-entropy (CE) supervised decoder. Existing continuous diffusion and flow-based language models match discrete LMs in performance but still depend on CE decoders because their flow trajectories are not guaranteed to end at valid token embeddings. ConvergeFlow addresses this by constraining the data predictor to the convex hull of token embeddings and training it solely with the mean squared error objective induced by flow matching. The authors prove that, under suitable regularity conditions, the resulting flow converges to valid token embeddings even when the data predictor has errors, enabling direct token prediction. They also introduce three sampling mechanisms that control the trade-off between generation perplexity and entropy. Experiments on OpenWebText show ConvergeFlow performs comparably to existing continuous and discrete diffusion language models.
Paper Overview
Research area: NLP
Authors: Na Li, Yuchen Jiao, Changxiao Cai
Published: 2025-08-26
arXiv: 2508.17626
Summary
Recent advances in continuous diffusion and flow-based language models (LMs) have achieved performance competitive with discrete LMs. However, existing continuous frameworks still rely on decoders supervised with cross entropy (CE), because the flow trajectories are not guaranteed to terminate at valid token embeddings.
ConvergeFlow addresses this limitation as an embedding-space flow-based language model that:
- Constrains the data predictor to the convex hull of token embeddings
- Trains solely with the mean squared error (MSE) objective induced by flow matching — no CE-supervised decoder required
Under suitable regularity conditions, the authors prove that the resulting flow converges to valid token embeddings despite errors in the data predictor, enabling direct token prediction without a CE-supervised decoder.
The paper further develops three sampling mechanisms to control the trade-off between generation perplexity and entropy.
Experiments on OpenWebText show that ConvergeFlow performs on par with existing continuous and discrete diffusion language models.
Links
- arXiv: https://arxiv.org/abs/2508.17626
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178634007