ELF: Embedded Language Flows — Kaiming He's MIT Team Builds a Fully Continuous Diffusion Language Model

Forum topic · 小凯 · 2026-05-13

Summary

ELF (Embedded Language Flows), from Kaiming He's group at MIT (arXiv:2605.10938), demonstrates that continuous diffusion language models can outperform discrete counterparts when the entire denoising trajectory stays in continuous embedding space. Discrete tokens are encoded by a frozen pretrained T5 encoder; a shared-weight transformer denoises in this space with Flow Matching (x-prediction), and discretizes to tokens only at the final step (t=1) via a learned unembedding — no per-step cross-entropy, no separate decoder. Training mixes 80% MSE denoising with 20% cross-entropy decoding, and Classifier-Free Guidance is imported from image diffusion using self-conditioning as the conditioning signal (training-time CFG). With only 105M parameters, 45B training tokens, and 32 sampling steps on OpenWebText, ELF achieves generative perplexity of 24, beating all discrete and continuous baselines including models trained on 10x more tokens. In conditional tasks, ELF surpasses autoregressive baselines on WMT14 De-En translation (BLEU 26.4 vs 25.2) and XSum summarization (ROUGE-1 36.0). Scaling from 105M to 652M parameters consistently improves the PPL-entropy frontier. Code is available on GitHub.

ELF: Embedded Language Flows — Deep Dive into Kaiming He's Team's Continuous Diffusion Language Model

> Source: arXiv:2605.10938 (MIT, Kaiming He's team) > Paper: https://arxiv.org/abs/2605.10938 > Code: https://github.com/lillian039/ELF

TL;DR

ELF (Embedded Language Flows) proves that continuous diffusion language models are competitive when the "continuous" design is carried through to the end. The denoising process stays entirely in continuous embedding space; discretization to tokens happens only at the final step (t=1) through a shared-weight network. With just 105M parameters, 45B training tokens, and 32 sampling steps, ELF reaches a generative perplexity of 24 on OpenWebText, beating all discrete and continuous diffusion baselines — including models trained on 10x more tokens.

Key points

Background: two routes in diffusion language modeling

Discrete DLMs (MDLM, Duo, LLaDA, Dream 7B) diffuse directly in token space and had been dominant.
Continuous DLMs (Diffusion-LM, CDCD, DiffuSeq, FLM, LangFlow) denoise in embedding space but historically lagged — because none were truly "continuous all the way": they applied per-step token cross-entropy, needed separate decoders, or used effectively discrete one-hot/simplex representations.
ELF's insight: keep continuous computation continuous and make discretization a single final step.

Core architecture

Training

Inference

Shared-weight network

Model sizes: ELF-B (105M, 768-d, 12 layers), ELF-M (342M), ELF-L (652M), with a 128-d bottleneck projection, RoPE, RMSNorm, SwiGLU, and QK-Norm.

Flow Matching details

Linear interpolation: z_t = t·x + (1-t)·ε with t ∈ [0,1].
x-prediction parameterization (rather than v-prediction); loss weighted by 1/(1-t)².
Training-time sampling: logit-normal (P_mean=-1.5, P_std=0.8).
Dual-branch training: 80% denoise (MSE), 20% decode (CE with per-token corruption rates); self-conditioning at 50% probability on the denoise branch.

Classifier-Free Guidance from image diffusion

Self-conditioning predictions act as the conditioning signal for CFG.
Training-time CFG avoids extra inference passes.
Control tokens: time (4), CFG scale (4), mode (4).
Conditional generation prepends clean condition embeddings, with 10% condition dropout and dual CFG.

Results

Unconditional generation (OpenWebText, generative PPL ↓):

ELF (SDE), 105M, 32 steps: 24 — beats MDLM (~45 at 1024 steps), Duo (~40), FLM (~50), LangFlow (~55), and all distilled variants (~28–32).
No distillation needed; trained on 45B tokens vs 500B+ for competitors.
Scaling: ELF-L with SDE reaches ~15, approaching autoregressive quality.

Machine translation (WMT14 De-En, BLEU ↑): ELF-B 26.4 vs autoregressive 25.2, MDLM 18.4, Duo 21.3, E2D2 24.8, CDCD 24.9 — a rare case of diffusion beating AR.

Summarization (XSum, ROUGE ↑): ELF-B achieves ROUGE-1/2/L of 36.0 / 12.2 / 27.8, beating AR (30.5/10.2/24.4) and all diffusion baselines.

Ablations

CFG scale ~2.5–3 optimal (quality/diversity trade-off).
Contextual embeddings matter: frozen pretrained T5 encoder (~24 PPL) beats learnable embeddings (~55) and random embeddings (~45).
SDE sampling significantly outperforms ODE at low step counts; they converge at 128+ steps.

Training setup

Optimizer: Muon, lr 0.002, batch 512; 80/20 MSE/CE loss mix.
Datasets: OpenWebText (seq len 1024, ~9B tokens), WMT14 De-En, XSum.

Unique positioning

only

Authors

Co-first authors: Keya Hu and Linlu Qiu (order decided by coin flip); corresponding author: Kaiming He (MIT).
Keya Hu: first-year MIT EECS PhD student (ACM class, Shanghai Jiao Tong University), co-advised by Kaiming He and Jacob Andreas.
Kaiming He: creator of ResNet (2015), joined MIT from Meta AI in 2024.

Implications and limitations

"Discrete vs continuous" is a false dichotomy: intermediate computation can live in continuous space with discretization only at output.
Image diffusion techniques (CFG, Flow Matching, SDE/ODE samplers) transfer directly to language.
Limitations: max tested scale 652M (no 7B+ yet), max sequence length 1024, wall-clock efficiency vs autoregressive inference not yet fully characterized.

References

1. ELF paper: arXiv:2605.10938 — https://arxiv.org/abs/2605.10938 2. Code: https://github.com/lillian039/ELF