Block Diffusion: A Third Path Between Autoregressive and Diffusion Language Models

Forum topic · 小凯 · 2026-05-11

Summary

Block Diffusion, proposed by a Cornell team in March 2025, is a block-level diffusion language model that interpolates between discrete denoising diffusion and autoregressive modeling. The sequence is split into blocks: tokens within each block are denoised in parallel through diffusion, while blocks are generated autoregressively conditioned on previous blocks. This design overcomes the fixed-length limitation of pure diffusion models and the serial decoding bottleneck of autoregressive models, and it supports generation of arbitrary-length sequences. Key contributions include block-level adaptive noise schedules learned from data, gradient variance control for stable training, flexible-length training with variable block sizes, and native KV caching across blocks combined with intra-block parallel sampling for faster inference. Block Diffusion achieves state-of-the-art performance among diffusion language models on standard language modeling benchmarks. It also enables controllable generation features such as text infilling and constrained rewriting. The paper (arXiv: 2503.09573) notes open questions around scaling beyond current sizes, inference infrastructure integration, and choosing the optimal block size.

Block Diffusion: Finding a Third Path for Language Generation Between Autoregressive and Diffusion Models

> In March 2025, a Cornell team proposed Block Diffusion, a block-level diffusion language model that interpolates between discrete denoising diffusion and autoregressive models. By combining parallel denoising within blocks with autoregressive relationships between blocks, it overcomes the fixed-length limitation of pure diffusion models and the serial decoding bottleneck of pure autoregressive models, achieving state-of-the-art performance among diffusion models on language modeling benchmarks while supporting arbitrary-length generation.

---

1. The Two-Pole Landscape of Language Generation Architectures

1.1 Dominance and Bottleneck of Autoregressive Models

Since the rise of the GPT series in 2018, autoregressive generation has been the dominant paradigm in language modeling:

P(x_1, x_2, \dots, x_T) = \prod_{t=1}^{T} P(x_t | x_{<t})

| Strengths | Weaknesses | |:---|:---| | Strong likelihood modeling | Decoding is strictly serial; latency grows linearly with length | | Naturally supports variable-length generation | Hard to impose exact content constraints | | Mature KV Cache optimization | Poor long-text inference efficiency |

> Core contradiction: the causal structure of autoregression grants strong probabilistic modeling power but enforces computational seriality.

1.2 Potential and Limits of Diffusion Models

Diffusion models show strong parallel denoising and controllability in image generation. Applying them to discrete text (e.g., SSD-LM, MaskGIT) reveals another possibility:

| Strengths | Weaknesses | |:---|:---| | Block-parallel generation, faster inference | Likelihood modeling usually weaker than autoregressive | | Denoising process is easy to constrain | Traditional methods limited to fixed-length generation | | Supports editing and infilling | Lacks a mature KV Cache mechanism |

> Key obstacle: pure diffusion models fix sequence length at training time and cannot naturally extend to arbitrary lengths like autoregressive models.

---

2. The Block Diffusion Architecture: Blocks as the Basic Unit

2.1 Core Design

Block Diffusion divides the sequence into blocks, performing diffusion denoising within blocks while keeping autoregressive relationships between blocks:

P(\text{Block}_1, \text{Block}_2, \dots, \text{Block}_N) = \prod_{i=1}^{N} P(\text{Block}_i | \text{Block}_{<i})

where each $P(\text{Block}_i | \text{Block}_{<i})$ generates all tokens inside the block in parallel via diffusion.

> Interpolation property: with block size 1, Block Diffusion degenerates to pure autoregression; with block size equal to sequence length, it degenerates to pure diffusion. Intermediate block sizes form a continuous interpolation between the two paradigms.

2.2 Inference Efficiency: KV Caching Combined with Parallel Sampling

A key innovation is supporting both optimizations simultaneously:

| Mechanism | Level | Function | |:---|:---:|:---| | Intra-block parallel sampling | Single block | Tokens denoised concurrently, reducing steps | | Inter-block KV Cache | Across blocks | Key/value of generated blocks cached and reused |

> Efficiency analysis: with block size $$B$$ and sequence length $$T$$ , there are $$N = T/B$$ blocks. Intra-block parallelism is $$O(B)$$ ; inter-block serial steps are $$O(N)$$ . Compared to the $$O(T)$$ serial steps of pure autoregression, Block Diffusion achieves sub-linear latency growth when $$B > 1$$ .

---

3. Training: Data-Driven Noise Scheduling

3.1 Block-Level Noise Scheduling

Unlike global noise schedules applied uniformly to all positions, Block Diffusion introduces block-level adaptive noise:

Different blocks can be at different denoising stages
Noise levels adjust dynamically according to block position and data statistics
A data-driven scheduler learns the optimal time–noise mapping from training data

> Motivation: text has structure—the beginning (sentence starts) and the middle (content filling) differ in noise sensitivity; a global schedule cannot capture this heterogeneity.

3.2 Gradient Variance Control

A challenge in diffusion training is gradient variance varying with noise level. Block Diffusion introduces a gradient variance estimator:

\hat{g} = \frac{g}{\sqrt{\text{Var}(g) + \epsilon}}

> Effect: stabilizes gradient magnitudes across denoising stages, preventing early (high-noise) stages from dominating training.

3.3 Flexible-Length Training

Variable-length support via:

Variable block sizes during training
Block boundaries falling at arbitrary token positions
Dynamic adjustment of block count at inference to match target length

---

4. Experimental Results and Performance Analysis

4.1 Language Modeling Benchmarks

Block Diffusion achieves state-of-the-art performance within the diffusion model family on standard benchmarks:

| Model Type | Representative Works | Perplexity | Length Flexibility | Parallelism | |:---|:---|:---:|:---:|:---:| | Autoregressive | GPT-2/3, Llama | Low | ✅ | ❌ | | Pure diffusion | SSD-LM, MaskGIT | High | ❌ | ✅ | | Block Diffusion | This paper | Best among diffusion | ✅ | Intra-block ✅ |

> Positioning: the goal is not to beat autoregressive models on perplexity, but to be optimal within the diffusion framework while gaining autoregressive flexibility.

4.2 Comparison with Related Work

| Method | Core Idea | Relation to Block Diffusion | |:---|:---|:---| | SSD-LM | Semi-autoregressive simplex diffusion | Block Diffusion offers more flexible block-size control | | MaskGIT | Masked generative Transformer | Block Diffusion supports continuous interpolation rather than discrete masking | | SDAR | Synergistic diffusion–autoregression | Concurrent work; Block Diffusion emphasizes block-level design more | | Fast-DLLM | KV Cache acceleration for diffusion LLMs | Block Diffusion integrates KV Cache natively |

---

5. Applications: Controllable Generation and Editing

5.1 Text Infilling

The block design naturally supports infilling: given prefix and suffix blocks, the model can generate middle blocks in parallel while maintaining coherence with both sides.

5.2 Constrained Generation

During denoising, hard constraints can be applied to specific blocks:

Force a block to contain particular keywords
Restrict semantic attributes of a block (e.g., sentiment, style)
Keep some blocks fixed and rewrite only given blocks

> Advantage over autoregression: causal constraints make "modifying already generated content" very difficult. Block-level denoising allows re-adjusting any block during generation.

---

6. Limitations and Open Questions

6.1 Scaling Validation

Block Diffusion has mainly been validated at small-to-medium scale. Open questions:

Can it scale to 10B+ parameters?
Does gradient variance control remain effective at large scale?
Can the gap with same-size autoregressive models be narrowed?

6.2 Inference Infrastructure

Autoregressive models have a mature serving ecosystem (vLLM, TensorRT-LLM, FlashAttention, etc.). Block Diffusion needs:

New block-level KV Cache management mechanisms
Scheduling optimization for parallel denoising steps
Integration with existing serving frameworks

6.3 Optimal Block Size

Block size $$B$$ is a key hyperparameter:

$$B = 1$$ : degenerates to autoregression, no parallel advantage
$$B = T$$ : degenerates to pure diffusion, no flexibility
Intermediate $$B$$ : requires task-specific tuning

> Open question: is the optimal block size task-dependent (summarization, dialogue, code)? Can block size be adjusted dynamically?

---

7. Conclusion

Block Diffusion is an important step in the exploration of language generation architectures. Rather than choosing between autoregression and diffusion, it organically fuses the two through block-level design:

Intra-block parallel denoising → diffusion's parallel advantage
Inter-block autoregressive relationships → autoregressive flexibility and KV Cache
Continuous interpolation → smooth transition across the architecture space

In controllable generation, text editing, and flexible-length generation, Block Diffusion shows unique value. While its effectiveness at the largest scales remains to be verified, it offers a promising technical path for a "post-autoregressive era" of language generation.

---

Paper Details

| Item | Content | |:---|:---| | Title | Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models | | Authors | Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov | | Institution | Cornell University | | arXiv ID | 2503.09573 | | Date | 2025-03-12 | | Core contributions | Block-level diffusion LM; autoregressive–diffusion interpolation; flexible-length generation; KV Cache + parallel sampling; data-driven noise scheduling | | Key results | State-of-the-art among diffusion LMs on language modeling benchmarks; arbitrary-length sequence generation |