Paper: Hierarchical Continuous Diffusion Language Models (HC-DLM)

Forum topic · 小凯 · 2026-10-04

Summary

HC-DLM (Hierarchical Continuous Diffusion Language Models) is a new approach that couples discrete token generation with a continuous latent trajectory inside a single principled denoising process, with a training objective derived from the variational lower bound on token likelihood. The method addresses structural bottlenecks in both discrete and continuous diffusion language models: discrete diffusion samples each token independently from its marginal during parallel decoding, severing dependencies among co-decoded tokens, while continuous diffusion denoisers lack any constraint tying the shared state to valid token configurations until final decoding. Unlike recent methods that attach continuous context to self-consistent discrete chains, HC-DLM makes latent variables the sole persistent generative state—tokens are read out each step and fed back as scaffolding for the next latent update. Experiments on structured reasoning (Sudoku), math programming (Countdown), and language modeling (LM1B) show that at equal model scale HC-DLM outperforms both discrete and continuous diffusion baselines, achieving higher puzzle accuracy and lower generation perplexity. Paper: arXiv 2610.02193; project page: https://hc-dlm.github.io/.

Paper Overview

Field: NLP Authors: Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing Published: 2026-10-01 arXiv: 2610.02193 Project page: https://hc-dlm.github.io/

Abstract (English)

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded.

Key Idea

To address this, the authors propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process. The training objective is derived from a variational lower bound on token likelihood.

Unlike recent approaches that append continuous context to a self-consistent discrete chain, HC-DLM makes the latent variables the sole persistent generative state:

Tokens are read out from the latent state at each step.
The decoded tokens feed back as scaffolding for updating the latent variables in the next step.

Results

At equal model scale, HC-DLM outperforms both discrete and continuous diffusion baselines on:

Structured reasoning (Sudoku): higher puzzle accuracy
Math programming (Countdown): higher puzzle accuracy
Language modeling (LM1B): lower generation perplexity

--- *Auto-collected on 2026-10-04.*