Summary
Diffusion Language Models (DLMs) are expensive at inference because they rely on iterative denoising, creating a need for effective pruning. Most pruning heuristics for DLMs are borrowed from autoregressive (AR) LLMs and assume that attention sink tokens should be preserved, since sinks act as stable global anchors in AR models. This paper challenges that assumption. The authors analyze attention-sink positions across the full generation trajectory and find substantially higher variance in DLMs than in AR models: dominant sink locations shift across denoising timesteps, indicating that sinks are often transient and less structurally essential. Building on this finding, they propose Sink-Aware Pruning, a retraining-free method that automatically identifies and prunes unstable sinks in DLMs. Experiments show that, under matched compute budgets, the approach achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines.
Paper Overview
Research area: NLP
Authors: Aidar Myrzakhan, Tianyi Li, Bowei Guo, Shengkun Tang, Zhiqiang Shen
Release date: 2026-02-19
arXiv: 2602.17664
Key Points
- Problem: Diffusion Language Models (DLMs) have high inference cost due to iterative denoising, motivating efficient pruning.
- Limitation of prior work: Existing pruning heuristics are largely inherited from autoregressive (AR) LLMs and typically preserve attention sink tokens, because AR sinks serve as stable global anchors.
- New finding: The assumption does not hold for DLMs. Across the full generation trajectory, the attention-sink position exhibits substantially higher variance—measured by how the dominant sink locations shift across timesteps—indicating that sinks are often transient and less structurally essential than in AR models.
- Proposed method: Sink-Aware Pruning, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs).
- Results: Without retraining, the method achieves a better quality–efficiency trade-off and outperforms strong prior pruning baselines under matched compute.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178208054