ConvergeFlow: An Embedding-Space Flow-Based Language Model with Provable Convergence to Token Embeddings

Forum topic · 小凯 · 2026-08-26

Summary

ConvergeFlow (arXiv:2508.17626, August 2025) is a continuous, embedding-space flow-based language model proposed by Na Li, Yuchen Jiao, and Changxiao Cai. Existing continuous diffusion and flow-based language models still depend on decoders supervised with cross entropy (CE), because their flow trajectories are not guaranteed to terminate at valid token embeddings. ConvergeFlow addresses this by constraining the data predictor to the convex hull of token embeddings and training it solely with the mean squared error objective induced by flow matching. The authors prove that, under suitable regularity conditions, the resulting flow converges to valid token embeddings even when the data predictor contains errors, enabling direct token prediction without any CE-supervised decoder. They also develop three sampling mechanisms to control the trade-off between generation perplexity and entropy. Experiments on OpenWebText show that ConvergeFlow performs comparably to existing continuous and discrete diffusion language models, offering a theoretically grounded route to decoder-free continuous text generation.

Paper Overview

Research Area: NLP Authors: Na Li, Yuchen Jiao, Changxiao Cai Published: 2025-08-26 arXiv: 2508.17626

Abstract

Recent advances in continuous diffusion and flow-based language models (LMs) have achieved performance competitive with discrete LMs. However, existing continuous frameworks still rely on decoders supervised with cross entropy (CE), because the flow trajectories are not guaranteed to terminate at valid token embeddings.

Motivated by this limitation, the authors introduce ConvergeFlow, an embedding-space flow-based language model, which:

Constrains the data predictor to the convex hull of token embeddings
Trains it solely with the mean squared error (MSE) objective induced by flow matching

Under suitable regularity conditions, the paper proves that the resulting flow converges to valid token embeddings despite errors in the data predictor, enabling direct token prediction without a CE-supervised decoder.

Key Contributions

1. Provable convergence: Theoretical guarantees that flow trajectories terminate at valid token embeddings, removing the need for cross-entropy supervision at decoding time. 2. Pure flow-matching training: Only the MSE objective induced by flow matching is used. 3. Three sampling mechanisms: Designed to control the trade-off between generation perplexity and entropy.

Results

Experiments on OpenWebText demonstrate that ConvergeFlow performs on par with existing continuous and discrete diffusion language models.

Source: arXiv:2508.17626