Paper Overview
- Field: NLP
- Authors: Aidar Myrzakhan, Tianyi Li, Bowei Guo, Shengkun Tang, Zhiqiang Shen
- Posted: 2026-02-19
- arXiv: 2602.17664
- Attention sinks in AR LLMs are stable global anchors, but in DLMs they are largely transient, with dominant sink positions shifting across denoising timesteps.
- Standard pruning heuristics that preserve sink tokens are therefore suboptimal for DLMs.
- Sink-Aware Pruning removes unstable sinks automatically, requiring no retraining, and delivers improved quality at matched compute versus prior baselines.
Summary
Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, motivating efficient pruning. Existing pruning heuristics largely inherited from autoregressive (AR) LLMs, typically preserve attention sink tokens because AR sinks serve as stable global anchors.
The authors show that this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps), indicating that sinks are often transient and less structurally essential than in AR models.
Based on this observation, they propose Sink-Aware Pruning, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs). Without retraining, the method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute.
Key Takeaways
*Auto-collected on 2026-06-24.*