ALoDLM: Diffusion LM Beats Its Autoregressive Teacher by Spending Compute on Hard Tokens

Forum topic · 小凯 · 2026-10-11

Summary

ALoDLM, from Amazon and University of Illinois Chicago, retrains Qwen3-8B as a diffusion language model using 5B tokens, and becomes the first same-scale diffusion LM to outperform its autoregressive base on average (80.3 vs 78.5 across 11 benchmarks). The paper diagnoses a compute-difficulty mismatch: standard diffusion models allocate equal computation to easy and hard tokens. ALoDLM adds a recurrent core between a Prelude and Coda, with an ExitGate letting confident tokens exit early while hard tokens keep refining their hidden states across up to four inner iterations—unlike prior confidence-based decoding, hidden computation is not discarded. Trained via a conditional NELBO with a variance-reduced gradient estimator, the model spontaneously learns lower exit probability (0.369) on numeric tokens. GSM8K throughput is ~2.7x vLLM-served Qwen3-8B at equal or better accuracy. Test-time scaling via the exit threshold yields +1.2 points at 46% more compute. Released Oct 2 (arXiv 2610.04198, amazon-science/ALoDLM), weights are CC-BY-NC-4.0, non-commercial.

ALoDLM: A Diffusion Language Model That Beats Its Own Teacher by Spending Compute on Hard Tokens

Source: arXiv 2610.04198 | Code: amazon-science/ALoDLM | 8B weights released Oct 2

The headline result

A team from Amazon and the University of Illinois Chicago took Qwen3-8B, ran a round of SFT on 5 billion tokens, and replaced the "predict the next token one at a time" objective with parallel infilling — the diffusion language model training target. Same architecture, different training: the average score across 11 benchmarks rose from 78.5 to 80.3. For the first time, a diffusion language model at the same scale outperformed its autoregressive base on average.

The problem: compute–difficulty mismatch

Diffusion language models (DLMs) predict many tokens in parallel, so they are inherently fast — but quality has trailed autoregressive models of the same size. The paper offers a blunt diagnosis: compute–difficulty mismatch. In infilling, some blanks are trivial ("Nanjing Yangtze River" → "Bridge") and some require real reasoning (a middle step of a proof). Existing diffusion models treat every blank identically, giving every position the same compute at every denoising step. Easy positions waste compute; hard ones don't get enough.

The method: recurrent core with an exit gate

ALoDLM splits the Transformer into three sections:

Prelude — embeddings plus a front block
Recurrent Core — the middle 16 layers in the 8B version, which are iterated
Coda — a final block that reads out logits

Within each outer denoising step there is an inner loop: representations pass through the core, Coda reads logits once, and an extra ExitGate head predicts the probability that a token is "ready to hand in." Low-entropy, high-confidence tokens exit early and become discrete context for neighbors still pending. Hard tokens don't restart from scratch — their hidden states are preserved and carried through the recurrent core for up to four rounds.

The key detail: prior confidence-based decoding also defers uncertain tokens, but the deferred tokens lose all hidden computation and are recomputed from masked embeddings — like a student tearing up scratch paper after each wrong answer. ALoDLM doesn't tear up the scratch paper. That's the essential difference between persistent hidden-state refinement and mere "delayed submission."

Training: exit timing as a latent variable

How does the model learn when to exit? It isn't taught explicitly. Exit timing is treated as a latent variable: the authors derive a conditional negative evidence lower bound (NELBO) that puts token prediction and compute allocation in one objective, stabilized with a variance-reduced unbiased single-trajectory gradient estimator.

The payoff is visible: on GSM8K, MATH-500, MBPP, and HumanEval, numeric tokens have a first-round exit probability of only 0.369 — 11.5% below the overall average. Nobody taught it "numbers need more thought." It figured that out on its own.

The scorecard, with caveats

8B version: average of 80.3 across 11 benchmarks, beating Qwen3-8B (78.5) and all tested diffusion models — LLaDA 53.3, Dream 60.5, Fast-dllm-v2 61.0, SDAR 74.2, WeDLM 75.1. But "beats on 11 benchmarks" needs unpacking: per-task it's ten wins and one tie — MMLU ties Qwen3 at 76.6, and the best MMLU score actually belongs to SDAR (78.5).

1.7B version: wins on average (65.5 vs 63.8) but loses individual tasks: MATH-500 by 10.2 points (61.8 vs 72.0) and MMLU.

Speed: ~2.7x GSM8K throughput vs vLLM-served Qwen3-8B — with the qualifier "at the same or higher accuracy." That qualifier isn't decorative; it's where such comparisons are easiest to cheat, and the paper stating it in the main figure caption is to its credit.

Depth: K=4 is the maximum recurrent depth set at training. K=8 is no better than K=4; K=2 clearly saturates. More depth is not better — mid-network cycling beats tail cycling.

Test-time scaling: a free knob

The exit threshold q controls how eager tokens are to hand in. Tightening it raises average per-token loop depth from 1.6 to 2.34 and lifts the 11-benchmark average from 77.9 to 79.1 — +1.2 points for 46% more compute. Not a spectacular slope, but the axis is free: same base, same weights, one knob.

By default the model uses only 1.6 rounds per token (cap: 4), meaning average compute is ~60% below "run every token to max" — and scores are still higher. Adaptivity isn't just reallocation; it's net savings.

Two threads on the main line

1. Compute follows difficulty. In agents this appears as allocating verification bandwidth by blast radius; in RL, as spending budget on hard tasks. ALoDLM lands it at token granularity: compute saved by early-exiting easy tokens flows entirely to hard ones. CodeHack's "judgment to the model, execution to code" is task-level division of labor; ALoDLM is per-step division within a sequence — different mechanisms, same ledger page.

2. The generation-steps spectrum. From autoregressive to diffusion to distillation to one-step generation, the trend has been "fewer steps." ALoDLM goes the other way: keep the steps, make compute within each step adaptive. Both roads converge on the same place — compute follows difficulty, not procedure.

Licensing and availability

ALoDLM-8B ships under CC-BY-NC-4.0 (non-commercial) — the Qwen3 base itself is Apache 2.0, but Amazon's release can't be sold directly. The amazon-science/ALoDLM repo was created Oct 2 with an optimized inference implementation; within six days it gathered 34 stars and 600+ weight downloads.

To see what it's really doing, check the exit-probability chart on the project page and find the numeric-token bar: 0.369. The paper's most honest number lives there.