Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast Diffusion LLM Inference

Forum topic · 小凯 · 2026-09-24

Summary

Flash-dLLM is a training-free inference acceleration framework for diffusion large language models (dLLMs) proposed by Quan Nguyen-Tri, Mukul Ranjan, and Zhiqiang Shen (arXiv:2609.26796). While dLLMs offer non-autoregressive text generation as an alternative to autoregressive LLMs, their deployment has been limited by the lack of effective KV caching and scalable parallel decoding. Flash-dLLM identifies GPU memory I/O as the dominant bottleneck in KV-cache-enabled dLLM inference and introduces an I/O-aware fused KV-cache kernel that reduces redundant memory movement. On top of this, it proposes a KV-cache-driven draft-and-verify decoding strategy in which the dLLM itself acts as both drafter and verifier, requiring no auxiliary model. The unified design preserves generation quality while improving speed and scalability to longer sequences and larger batch sizes. On math reasoning and code-generation benchmarks, Flash-dLLM achieves 5.1x and 11.0x speedups over the previous strongest baseline Elastic-Cache on GSM8K and HumanEval respectively.

Overview

Field: LLM inference acceleration
Authors: Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen
Published: 2026-09-22
arXiv: 2609.26796

Abstract (full translation)

Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive LLMs by enabling non-autoregressive text generation. However, their practical deployment remains limited by inefficient inference, largely due to the absence of effective Key-Value (KV) caching and scalable parallel decoding mechanisms. Existing acceleration methods typically study KV caching and parallel decoding in isolation, overlooking the I/O bottlenecks that arise when cache reuse and parallel token verification are jointly applied.

In this work, the authors introduce Flash-dLLM, a training-free inference acceleration framework for fast and memory-efficient dLLMs. Flash-dLLM first identifies GPU memory I/O as a dominant bottleneck in KV-cache-enabled dLLM inference and addresses it with an I/O-aware fused KV-cache kernel that reduces redundant memory movement. Building on this optimized cache mechanism, Flash-dLLM further proposes an efficient KV-cache-driven draft-and-verify decoding strategy, where the dLLM itself serves as both drafter and verifier without requiring an auxiliary model.

This unified design enables faster decoding while preserving generation quality and improving scalability to longer sequences and larger batch sizes. Extensive experiments on mathematical reasoning and code-generation benchmarks demonstrate that Flash-dLLM consistently outperforms existing state-of-the-art dLLM acceleration methods in both inference speed and memory efficiency. In particular, it achieves $5.1\times$ and $11.0\times$ speedups over the prior strongest baseline Elastic-Cache on GSM8K and HumanEval, respectively.

Key Points

GPU memory I/O is the dominant bottleneck in KV-cache-enabled dLLM inference; a fused I/O-aware KV-cache kernel reduces redundant memory movement.
A KV-cache-driven draft-and-verify decoding strategy lets the dLLM serve as both drafter and verifier—no auxiliary model needed.
Training-free design preserves generation quality while improving scalability to longer sequences and larger batches.
Results: $5.1\times$ speedup on GSM8K and $11.0\times$ on HumanEval versus Elastic-Cache.